24 hours feat: #zapathon spike

PV. Thanks for zaps everyone as always! And big thanks to the “Zap! PV💜🧡” zapper :)

Last point as posted to GitHub too with a few extra points.

If you want explicit content - start an explicit only relay. The far majority of relays will be 99% general consumption.

However client apps lack private relay connection lists today. As apps to support them - and start a relay.

If you want some explicit content network on Nostr - just submit a NIP with a new kind. And you can spec it however you want. It can be perfected for indexing porn content or whatever you like.

I think anything else pushed into kind 1 is overkill. Kind 1 is for short messages, not a media gallery.

** and to clarify.. “you” wasn’t used to refer to anyone specific.

A limitation today is clients not being easily able to tag a new event with a content-warning. Could be due to App Store approval concerns. I’d start understanding that first.

Personally I think in depth pre-defined tags are the wrong approach. Net nanny and FWs use them, and it’s often inaccurate (e.g. lots of false gambling flags) - it encourages opinionated blacklists. My only exception is what would normally be under NSFL - almost always watching death occur.

I see normal hashtags as being flexible enough if you want some descriptors for searching or knowing what to expect before you un-blur media content. Explicit websites seem to just list tags. And Nostr events can be tagged without hashtags in content.

We are missing an explicit key in the kind0/profile metadata today. A profile image or banned could be explicit - not just their events. I think that’s easy to add - just add content-warning to the kind0 json. An edge case is showing an event before a profile event first loaded - you could see an explicit profile icon, if the event is not tagged (if it is tagged, and profile isn’t known, it could just blur by default).

An obvious issue is also different language. Which to use in events, how to handle typos, and so on.

And as always, it’s best effort as we have no guarantee the publisher will tag or use some pre-defined list. Reporting can be abused for censorship. For open networks I advocate better client filtering tools - rather than protocol level rules.

Apart from having some threads not load their parent recently, I don’t even notice Nostr is decentralised.

More scaling issues will come.. but we’ve effectively near seamlessly replaced closed with open, and noise with signal.

GM ☕️

Sure, thanks. ⚡️

Yep. The gossip model was effectively my initial approach too, for an unreleased relay recommendation engine, before I realised it was more complex. My approach was data driven, so basically which relays saw a pubkey’s 200 most recent events, rather than based on the kind 10002 pubkey relay manifest approach.

Basically similar to what you mentioned above, with a N coverage threshold and M max desired relays.

https://en.wikipedia.org/wiki/Set_cover_problem

Done. In purple.

It’s deceptively complex to do it optimally - but good enough works ok too. Which relays you receive from differs, even by kinds you want, and publishing is similar complex.

Not sure if helpful, but a dump of my notes below around relay selection. Trying to end up with a diagram and list of functions that explain what logic to apply when and how to pick and prioritise relays.

Paid relays, rate limiting, kind, destination pubkeys, your preferences, relay hints, proof of work, parent events, mentioned pubkeys, etc — all can factor in. Relay health/connectivity.

——

getBestOutboundRelayForEvent

(Note: A paid relay filter should apply unless you have paid)

Kind 0/3/10_002

Maximally broadcast

Publish to Metadata Indexer Relays

Note (without reply or mention)

Do you want to maximally target your followers?

Do you want to maximally broadcast?

Reaction

Include parent relay hints (if provided)

getBestOutboundRelayForPubkey(author)

Repost

Include parent relay hints (if provided)

getBestOutboundRelayForPubkey(author)

Do you want to maximally target your followers?

Do you want to maximally broadcast?

Reply

Include parent relay hints (if provided)

getBestOutboundRelayForPubkey(author)

Do you want to maximally target your followers?

Do you want to maximally broadcast?

Reply + Mention

Include parent relay hints (if provided)

getBestOutboundRelayForPubkey(mention) for each mention

Mention

getBestOutboundRelayForPubkey(mention) for each mention

DM

getBestOutboundRelayForPubkey(recipient)

Event Report

? Broadcast or targeted relays ?

Long form Content

? Broadcast or targeted relays

(Nostr-connect)

?

(Zap Service Provider)

?

getBestOutboundRelayForPubkey

Kind 10_002

NIP05

Kind3 (or EOL?)

Fallback to my Publish/Write relays (as they can query them for lookup)

----------------------------

getBestQueryRelayForPubkey

getBestQueryRelayForEvent Id

If only an event Id is known.. Shotgun query / your read relays

getBestQueryRelayForEvent

Include event relay hints (if provided)

Check parent event (if known) relay hints

getBestQueryRelayForPubkey for each known thread participant

Other considerations

Min POW Required

Rate Limiting

Thanks. It’s a few months old now, before browser extensions and relay AUTH.

I chose the chatbot approach to prevent DDOS, and downloading others people’s events - mostly DMs and deleted events. It works well, and is backed by my event driven backend - so adding more bots is easy.

And lastly.. here is a timeline of the original event. Deletion isn’t shown.. maybe I should add :)

Also. From around 80-90 relays being aggregated, here is which relays at least saw your deletion event.

https://api.nostrgraph.net/beta/events/9d0da9ce8d33da90a55cb8a7598c7f7b579093e9c945a11f7d6812fafacb29b7.json?pretty=true

Yep. Shared it somewhere in this thread.

It has the mutiny blaster relay in the default list. In practise I’ve found it doesn’t reach 300 relays.. it’s closer to 30-40. Maybe rate limiting, ip blocks, dead relays, not sure specifics.

#[8]

Here come the #zapathon

Not sure if you’ve seen this. I wrote it recently for a bounty.

https://cdn.nostrgraph.net/public/delete.html

The GitHub version has a couple extra features to load your relays. https://github.com/blakejakopovic/nostr_delete

This relay spam filter is still working awesome. Just checked my spam DLQ and so much new spam content that I’ve never seen - because the ML model that’s a couple months old is working great still. I do have other rules too.

The Redis spam stream is a rolling 15 minutes. That’s 256,436/hour or 70/second spam Nostr events. Before de-duplication as I filter them before that happens.

https://github.com/blakejakopovic/nostr-spam-detection

Nostr Excel day 2 - load a single event or a batch of events into rows and columns - in Excel Online ;) #[0]

Streaming a REQ works too 🤯

Just need to overhaul the functional interface and can release.

What kind of throughput do you need to handle?

I’m at around ~60 spam events/second and ~30 non-spam (rough averages). I was handling 250/second for bursts/importing events easily. Basically I process 10X the unique events (after spam removal). Something like 8-10 million events/day (with spam and dupes).

I’ve got two validation workers (I tried six at one point, but didn’t need them), four persist workers, and basically a Redis steam setup. Validation worker idle time is 1ms average (it batches). And persistence averages around 2-4ms. Validation worker also calls spam ML model.

Streams helped a lot. Batching does too. Deferring work as well - I process tags, meta events, zaps, contact lists , db usage, pow, relay meta, all after the fact - but a relay doesn’t need that post processing.

I also don’t broadcast any events until they have been persisted and validated. Mostly so I can de-dupe too.

I actually stopped connecting to Damus (for aggregation) a little while back due to spam volume. Basically other relays like nos.lol have the same data but filtered :)

Happy to chat more. What I have has been pretty low touch for a while now. Has limitation around broadcasting older events if workers offline for a while or importing missed events - but can address or filter as desired.