I want to send a Merkle Proof as a relay filter.
Send me everything that passes these filters that I don't already have on the phone.
Is it feasible?
Discussion
Not quite sure what you mean.
Reduce network usage. Instead of downloading the same duplicated info from 20 relays, we can download it from the fastest relays first and ask the others to send everything that is not included in the Merkle tree.
I'm not seeing how a merkle tree helps here is what I'm saying.
Are Merkle trees used in Nostr protocol? :-O
I first read your last post about trees and thought you were describing how you optimized. But after reading all the correspondence, I realized that you offer to use it.
It’s a good idea, but bloom filters would be more efficient. I give you a bloom filter that represents all events i have, and based on that you can send me back all events you know aren’t part of my bloom filter. The reason I’m suggesting bloom filters is because they’re much more space efficient.
I would extend it, that if you filter an already known note out, you could send a 1 of 0 if you have it without sending the note itself, so we will still know which notes are stored on which relays.
i believe this is how strfry relays handle syncing with other relays
https://github.com/hoytech/strfry#sync
`uses the yesstr protocol and performs a merkle-tree set reconcilliation against the specified relay.`
How heavy this is? Any chance of turning their implementation into a NIP and add into the regular filter options?
yeah ideally this would be a NIP for clients to efficiently deduplicate data from relays. it’s actually pretty efficient, and #[5] seemed to be speaking highly of the performance when relay.damus.io was switched over to strfry.
Correct me if I'm wrong but this seems to add overhead for no reason versus just sending the event IDs of what you have. Why add the overhead of the tree unless you want proper inclusion proofs (which would require merkelizing all events you have as a relay).
The hope was to tmget the tree or a bloom filter to reduce the amount of IDs. Amethyst generally has 50,000 IDs in the local database, which would make it a 3mb upload at every new filter request to a relay.
Merkelizing IDs though is just going to add more data, or if you dont send the whole tree add resource demands for relays to hash the tree from inputs.
How Merkel trees helps here? Merkel trees helps to identify if an item belong to it, but I don't see how it will help to fetch results
Put all the IDs you have in the tree. Send the tree to the relay. Now give me everything that i do not already have.
Is there a proposal for it already? Would love to see how that performance works on the relay side, I guess it would be an inverse proportion with the IDs array size
Certainly the date ranges filtering we have nowadays are insufficient on a network where the consensus points to zero. And there is no much proposals for relays on that sense.
#[2]
It looks feasible to me. Relay nodes will act as a miner to generate Merkel tree and broadcast. Low proof of computation as not much is at stake.
I was thinking the phone generates the tree and sends it to the server. The server simply checks if the id is in the tree or not.
Am I missing something?
I thought phones requesting a block of notes would be more efficient. Even if the node doesn't have it all, the phone/node knows how to complete it by requesting sub trees.
But let's say the phone already has 95% of the block and needs only the extra 5% (the phone doesn't know how much is left) because there was some type of connectivity issue.
The majority of the time, the phone already has everything. We just need new things and the things that were missed due to some connectivity issue or busy relay.
The first time a client subs all your follows to all your relays demands a lot of bandwidth because the client has no information about the status or location of notes.
But if the client remembers which relays first served notes from a particular pubkey and stored that in a table… on ALL SUBSEQUENT subscriptions of the session the client could just ask the fastest relay for the notes of a each pubkey to provide the notes for that particular pubkey.
This would reduce the bandwidth by a factor of n relays.
Wouldn't a bloom filter be better suited for that? (i am just guessing here)
What does that mean, exactly?
"Give me everything that i don't already have" query.
Great for relay backups :)
Would require relays to all have the same events and agree on their order (like a blockchain).
I believe a bloom filter is more in line with what you’re envisaging.
Checkout how the bitcoin minisketch proposal works , was talking about this today at lunch with #[2] #[3] and #[4]
#[4], didn’t know that was you. Followid.
Instead you should cache profile pictures and implement the Gossip model for fetching data efficiently.
