🤷🏻♂️ Sounds like we can stuff it in mempool now.
I get the desire for a nice system that has great answers for the latest changes, but they come with significant costs as well. I prefer and argue for a systems of independent, near-complete caches over a fancy network that knows where something lives.
Apparently the Damus relay had five hundred gigabytes of data – which means I could fit it all in RAM, twice. It's important to watch how constraints change over time
Good for you, not sure how is that relevant for rest of us and our phones, should we all query your RAM? Or who should we query?
The system I'm arguing for, you have the option of privacy. In a DHT, you have no such option
Says who? Who is preventing you from scrapping the DHT and build a push system or share what you downloaded on a USB stick?
I did my best to explain why you can't have the features that a DHT offer any other way, I never said you shouldn't do more on top.
Also, privacy can be achieved in other ways.
It seems you weirdly want to download the entire web and read it locally because that is awesome and private, sure, do that.
My question remains; what about phones and people who don't have 500 GB of RAM or storage at all? How should they find the data they are looking for?
If someone in their community has a full copy – they can ask them, and that's as far as the query needs to go. Or, you can keep asking less trusted servers until you find your answer. If you're in a conflict zone and you want to find someone controversial, this is important.
But no one is going to let you scrape the DHT to provide this kind of plausible deniability because query scraping is the kind of spam the DHT is supposed to keep out. The only way to get a full, private copy is if the system is designed around the efficiencies you gain from putting things in chunks
You are just mistaken. DHTs don't stop you from what you want to achieve as futile as I think it is, but the DHT doesn't make it any harder.
My relay has thousands of packets, nothing can stop me from gossiping with other relays, I just think it is a bad idea.
How would I get someone's record without revealing who I'm looking for?
Same way you plan to do it in your system, build a gossip network on top to share the scrapped data from the DHT or make a network of relays that forward things to the DHT but also gossip it with whoever wants to download it all ... or whatever.
There is nothing in using the DHT that makes using gossip in parallel impossible... the only thing makes gossip impractical is that it is inherently impractical.
Remember... Nostr already doesnt have gossip and actually started as a critique of gossip (Secure scuttlebutt), and Negentropy was invented on top.
Yet somehow you make it sound like Nostr relays are inherently gossip friendly, well, extend the same grace to Pkarr relays.
You get exactly what you want, but also a DHT to fall back to when your system inevitably fails for the vast majority of web needs.
Nostr relays aren't "gossip friendly", they just have a "latest" query. How do I ask the DHT for the latest updates?
Again, either scrap the dht as the BEP i linked to explains, or run Pkarr relays and add extra endpoint to get latest submitted stuff.
Also, what Nostr relays are you asking for latest updates? All of them?
DHT nodes organize themselves in a structural way so you can find them all if you try, so they should be simpler to scrape than Nostr relays, which you don't know them all.
Anyways, I think you are not suggesting a practical solution, and what you actually want is just an extra requirement that DHTs don't make harder at all.
