Playing with Voyage. I like it a lot so far. Although I miss the memes, the minimalistic vibe is amazing.
Thinking how to integrate it with some Lora based mesh (like Reticulum) to have a local city district relay to completely bypass internet. It would need much more compression though.
It should be integrated first at relay level probably and then local mesh node would do a relay interface.
#Reticulum does compression at the transport layer, so we don't have to think about it.
Really like this plan.
IMO, Voyage does need zaps and multiple reactions. Your thoughts?
I don't miss these features too much.
I mean more radical compression. Like don't transfer id, it can be calculated from other fields. Shorten the tags and Json fields of notes. you really need a few bits for things like "sig", "content" names.
Also really requesting and downloading only notes we don't have could be quite radical saving.
With Reticulum the hardest part would be to solve broadcast. We don't want to transfer end to end, every node seeing the note should save it to their local relay, which is a bit difficult with Reticulum.
Requesting and downloading only notes we don't have would make a big difference.
Although, Reticulum can do one-to-many broadcasts, like UDP and torrenting, which could reduce bandwidth enormously too, but obviously in an incompatible way.
Is there a good way to combine these approaches?
OK, so we can do rotating bloom filters.
Architecture:
We are syncing relays (which can be on a slow network and sync while we sleep). Let's say we have clever filters, so we don't sync everything (relay knows about follows and replies), but this is not that important, maybe for small networks, we can sync everything.
Algorithm:
- client relay creates a bloom filter with nonce (just a random number that gets applied to all hash functions in the filter construction). It sends filter ("give me notes for the past 24 hours"), nonce and the bloom filter of note ids matching this description it already has.
- server relay (or relays, we'll see if we can somehow broadcast this without overloading the network) collects the events. It fills a preefined number of packets with content that matches the filter and does not match the bloom filter. If there's nothing, it sends back "you've seen all" message (this can be trusted, bloom filter does not have false negatives).
- after receiving or after some time, the process is repeated with a different nonce.
The bloom filter for sure filters out note IDs that the client already has. It might filter out some other note IDs, but this is mitigated at the next request, because the bloom filter will have different false positives, because of different nonce.
Now we need the broadcast protocol for Reticulum and let's get coding. Since all reticulum nodes right now are basically Linux machines (RNode needs control), everyone can run their own relay, listen to wire traffic, update notes from the air and sync from time to time on everything it missed.
We will use unchanged relay software, unchanged Nostr clients (Voyage is perfect for this), only the syncer is required.
nostr:npub1f49twdlzlw667r74jz6t06xxlemd8gp2j7g77l76easpl8jsltvqvlzpez any inputs?
Of course we need hardcore wire compression of the notes, but that can be developed over time.
Quickly hacked encoding:
Original size: 600 bytes
Encoded size: 445 bytes
Encoded and compressed size: 292 bytes
Currently uses:
- Don't transmit ID if it can be correctly calculated (should be).
- JSON keys are replaced with integers, don't transfer string "created_at", when I can transfer "4".
- Run brotli on the data.
Encode hex strings as binary:
Original size: 600 bytes
Encoded size: 285 bytes
Encoded and compressed size: 289 bytes
178MiB 0:09:31 [ 318KiB/s] [================================================>] 100%
Processed 214488 events
Total original size: 182393202 bytes
Total encoded size: 125937545 bytes
Total encoded and compressed size: 96236760 bytes
Total compressed but not encoded size: 116194898 bytes
Average original size: 850.37 bytes
Average encoded size: 587.15 bytes
Average encoded and compressed size: 448.68 bytes
Compression ratio: 52.76%
Compression ratio without encoding: 63.71%
Here's a sample compression. It is slow, but Lora is much slower (although there are multiple compression and uncompression steps in this, which will not be necessary).
Anyway, what does this mean? We get events with average original size of 850 bytes (I got sample events from relays, quite a lot of them). Just encoding saves a lot. Compression is good, but if we are compressing without encoding, we only get 63.71% (with encoding 52.76%). That might basically be just stripping hashes (id).
If we compress a huge set of events, the difference is not so high:
125937545 1 sep 00:24 events.encoded
70426465 1 sep 00:24 events.encoded.br
186704164 31 aug 23:38 events.jsonl
81979371 31 aug 23:38 events.jsonl.br
Brotli creates a dictionary (I do it manually) and it's pretty good at it. The problem is when we want to transfer just one or a few events, we might not have that luxury.
Anyway, I am pretty satisfied with compression.
I would just limit the size. Fixed packets makes things easier and forces brievity.
Also, there is just one RF half duplex channel. So if the relay need to transmit Alice's messages, it can't receive new ones I believe.
Nor can anyone else as the channel is used.
I haven't looked at it closely but in any case my intuition says keeping messages very short will lead to better results.
Yes, keeping them short so they fit in one packet is a good idea. Or burst a few packets and then wait until user requests new bloom filter.
The point of the bloom filter is to make sure the sending relay does not send events the client already has.
Curious to know what you think about building a custom khatru relay
Is this about mesh networks, sorry I'm retarded.
cc nostr:nprofile1qqsxyllxr558rfkrn7df69cshgd3qy0rwev0a4l5sfuq754vpu5spjspz4mhxue69uhhyetvv9ujuerpd46hxtnfduhsz9nhwden5te0daexzmn8v4ehjmnr9e6x2cmg9uq3samnwvaz7tmwdaehgu3wvd5xz6tdvyhxjmnxduhsawyasv
