How much do we care about relay query speed vs note retention?
On a big aggregator like filter.nostr.wine, I could probably improve performance significantly by only serving the last 3 months of notes instead of our full history. Worth it?
Discussion
query > retention
It seems like it should be possible and pretty easy to achieve both:
- if one queries for a specific note, it shouldn’t depend on their amount
- if one queries for the events of last 3 months - it should use fast and small “latest 3 month events” index
- if one queries for older events or for not specified time range - it will be slower but still working
Or are you talking about some other use case?
Has to do with overall size of the db and my ability to scale horizontally vs vertically.
Personally, this is the value proposition of having both aggregators (y’all) and smaller solo servers (like mine) on my list.
Yall are the performant way to get data from a lot of places, mine is a permanent home for notes.
I’d have no issue if yall did a 90 day retention to speed things up.
Thank you. This is how I view it as well but wanted to get some opinions.
Much easier for us to retain your notes longer on nostr.wine than on filter.
I agree. And with broadcast=true, those notes are also persisted to the other relays. It’s not like youre talking about deleting content so it’s not there.
True, though the truth is that many large public relay operators wipe their DBs at will. It’s not reasonable for anyone to really expect them to retain notes.
What would this mean for the older notes? Is there a way to backup/search for these?
I would presume most (all?) filter.nostr.wine users also have nostr.wine on their relay list so they would still have their full history there.
We could certainly build tools to allow backups but there are already a few good ones out in the wild!
We could also offer an additional relay if there’s interest.
Something like archive.nostr.wine that is slower and has less readers but retains old history.
Seems like a good bridge. Not that anyone would need access to the archives all the time, but every now and then when you want to access that old note, it’s just good to have the options. Seems like that will be the way to maintain balance between memory and speed 🤙💜
Yeah!
Yes.
Archiving seems like an entirely different job. Making relays performant seems far more important.