there has to be spiders and search engines

ideally it minimises spidering usage by multiple indexes sharing data with each other

this is going to be essential, and also why having LLMs trained on this spidered data will be essential, and it needs to be compounding, so every node can take in new data and it modifies their model and updates for the current state

there is also the possibility that you make nostr-hooks so when you push stuff, it goes to relays that aggregate git activity events, and you don't even need spiders

that's something we can do that they can't do