Reduce network usage. Instead of downloading the same duplicated info from 20 relays, we can download it from the fastest relays first and ask the others to send everything that is not included in the Merkle tree.
Discussion
I'm not seeing how a merkle tree helps here is what I'm saying.
Are Merkle trees used in Nostr protocol? :-O
I first read your last post about trees and thought you were describing how you optimized. But after reading all the correspondence, I realized that you offer to use it.
It’s a good idea, but bloom filters would be more efficient. I give you a bloom filter that represents all events i have, and based on that you can send me back all events you know aren’t part of my bloom filter. The reason I’m suggesting bloom filters is because they’re much more space efficient.
I would extend it, that if you filter an already known note out, you could send a 1 of 0 if you have it without sending the note itself, so we will still know which notes are stored on which relays.
This is what I was thinking it may be possible. With filters you can subscribe to a relay for “all notes Not included in a list” so instead of asking all relays to send the same, I can ask it to the first and then check the rest
#[0]
That said, there is a latency risk I guess as my mobile app would need to process the notes from first relay first, create a new subscription and only then go check the next relay instead of asking all data to everybody at once
Since many relays take seconds to reply, I do this "process relay results, create new subscriptions" already in Amethyst. It's probably not helping slow relays, but it at least helps with network use.
i believe this is how strfry relays handle syncing with other relays
https://github.com/hoytech/strfry#sync
`uses the yesstr protocol and performs a merkle-tree set reconcilliation against the specified relay.`
How heavy this is? Any chance of turning their implementation into a NIP and add into the regular filter options?
yeah ideally this would be a NIP for clients to efficiently deduplicate data from relays. it’s actually pretty efficient, and #[5] seemed to be speaking highly of the performance when relay.damus.io was switched over to strfry.
Correct me if I'm wrong but this seems to add overhead for no reason versus just sending the event IDs of what you have. Why add the overhead of the tree unless you want proper inclusion proofs (which would require merkelizing all events you have as a relay).
The hope was to tmget the tree or a bloom filter to reduce the amount of IDs. Amethyst generally has 50,000 IDs in the local database, which would make it a 3mb upload at every new filter request to a relay.
Merkelizing IDs though is just going to add more data, or if you dont send the whole tree add resource demands for relays to hash the tree from inputs.