Replying to Avatar ᴛʜᴇ ᴅᴇᴀᴛʜ ᴏꜰ ᴍʟᴇᴋᴜ

i will make this in my rewrite of realy too... it will be a relay side thing, a separate "sync" endpoint that will have a thread maintaining the IDs of recently stored events in a cache and per-subscriber queue state management that will always send events, and the receiver will ack them instead of the current scheme where it's fire and forget
Avatar
ᴛʜᴇ ᴅᴇᴀᴛʜ ᴏꜰ ᴍʟᴇᴋᴜ 6mo ago

semi started me thinking towards this direction when we came up with the idea of creating an endpoint that allows a client to know the internal sequence number of events, as this allows pull side queuing, but i think push side queuing would work even better.

with just this one feature added, you can have a whole cluster of relays all keeping up to date with the latest from each other, with multiple levels of propagation as well as bidirectional so for example two relays can stay in sync with each other in both directions, this also requires extra state management so that they don't waste time sending events to subscribers that they got from subscribers in the other direction.

Reply to this note

Please Login to reply.

Discussion

Avatar
ᴛʜᴇ ᴅᴇᴀᴛʜ ᴏꜰ ᴍʟᴇᴋᴜ 6mo ago

the other thing that is required also is that relays need to have configurable garbage collection strategies, so that you can have master/archival relays which get huge storage, and smaller ones that prune off stuff that have stopped being hot items to contain their utilization, so, archive relays and cache relays.

and then, yes, you further need a model of query forwarding so a cache relay will propagate queries to archives to revive old records, the caches could allocate a section of their data that is just references to other records, stored with the origin of the original, now expired event, that also is maintained within a buffer size limit, so they know exactly which archive to fetch it from.

lots of stuff to do... i started doing some of this with the original "replicatr" my first attempt at a nostr relay, implemented a whole GC for it, wrote unit tests for it... the whole idea was always about creating multi-level distributed storage. unfortunately no funding to focus on working on these things, instead i'm stuck building some social media dating app system lol