Subnostr

I’ve read a lot of your and others long form posts on DVMs. I understand the ideas and philosophy but again you’re losing me on the actual implementation details.

Step 3 as you’ve described it, is just an aggregator. Nobody can do this on the fly for text search. The operators that have this data are going to be large DBs who have been aggregating nostr events from many relays. These happen to be the exact same providers/relays that already offer NIP-50.

You’ve doubled the amount of steps and still you’re going to the same sources for information. Search is a function that is most useful when provided by aggregators.

That DVM search implementation is many times more complicated (and slower) than adding a “search” REQ filter. That’s the comparison I’m making.

hodlbod 1y ago

You're right that latency is not great, and on the fly indexing might not be feasible. Maybe a direct connection, bypassing DVMs and returning actual events would be a better interface. That's another implementation detail we can add later once the DVM model is proven.

The thing about using relays as aggregators is there's currently no way to know if they have the events you want. I drafted a NIP a looong time ago that attempted to solve this problem using relays: https://github.com/nostr-protocol/nips/pull/259. The idea is that a relay would be able to recommend another relay that indexes its content, and which supports the functionality the client is looking for. So small relays could still self-host, but rely on big indexers to serve search. Otherwise, clients still have to rely on centralized services like nostr.band and nostr.wine to find the notes they're looking for.

This is particularly bad with relays that enforce AUTH, since there's no way for an indexer to build an index for content exclusively on those relays. Authorized indexers (indexers that are authenticated with these private relays) are the only way to expose that data other than making relays build every search/discovery feature directly into the relay interface. So if a client wants to search/analyze data from a particular relay, they need to know who is indexing that relay. DVM requests allow clients to specify the relays they want to look at, which DVMs can answer or not depending on whether they index the target relays.

I'm not saying DVMs are integral to this process, just that they're the only things that exist that can support separating data and functionality right now. It would be fine to have a "search relay", or an "analytics relay" and for the NIP 11 document to point clients to themselves for search/analysis, or to some other relay for search/analysis.

A succinct way to describe my goal is: there should be no stranded data in nostr. All of it should be indexable and discoverable to the extent the custodian wants it to, however small the relay instance. NIP 50 search built into nostr.wine doesn't solve this.

Reply to this note

Please Login to reply.

Discussion

Mazin 1y ago

I know we are discussing DVMs more broadly but I want to focus on search (and I believe topic classification fits in to the same category).

For search, you generally don’t know what events you want so I don’t think not knowing if an aggregator has particular events or not is really a concern. If you don’t find what you’re looking for, you can always look on a different aggregator. Better yet, connect to a handful of them and execute your REQs in parallel. If you don’t find them on any of the aggregators, I don’t see why you would be able to find them on a DVM.   Relays that enforce AUTH for certain or all events generally do NOT want their events to be indexed. Creatr.nostr.wine certainly doesn’t want public indexers involved. If we wanted to offer search for our patreon relay, we would offer it directly on the relay - that makes way more sense than authorizing certain aggregators to access the data.   There have been on and off discussions about implementing NIP-50 in to strfry. A few relay implementations supporting it out of the box would decentralize search a lot more overnight. I’m afraid I still don’t see any reason to involve a DVM here.

Mazin 1y ago

On a separate but related note - I think we can expand NIP-50 to offer a lot more optional functions in the search, including one you mentioned.

Isolating results to only one relay (even if NIP-50 relay is an aggregator), negative keywords, wildcards, topics, language, sentiment, nsfw, etc etc.

Obviously I don’t expect every relay to implement these, but even if we only have a handful of aggregators we can query directly, thats at least on the decentralization spectrum compared to all existing social media search.   We should also expect the number of aggregators and search providers to grow with time if nostr succeeds (even if their data become more fragmented).

Mazin 1y ago

One last thing and then I’ll stop harassing you guys, just lots of questions and want to understand.

We DO have common ground here. I think DVMs are useful, particularly for things like translation, text to speech, etc. The more generalized the request can be and the less coupled it is to a relay, the more sense it makes to me to use a DVM. In those instances it’s incredibly valuable to have multiple service providers accessible through a single job request on any client/relay.

When the providers of the data are relays, as in the case of NIP-50, I don’t see why we should bend DVMs to try to fit this use case.

Happy Sunday!

hodlbod 1y ago

You as well! I think maybe a part of our disagreement is our vision of how big relays ought ultimately be. I would like there to be many small relays which are cheap to operate and sovereign, but which can delegate advanced functionality to other services. This doesn't make sense for nostr.wine, because you have no problem running those other services. Both visions can coexist, but as a client dev I'd like to access both with the same interface.

But these are just opinions, the proof is in the pudding. I look forward to seeing what you come up with. 🫡