Relay specialization to achieve optimum functionality
V2
Requirements:
Noone misses out on any event
Efficiency / lower bandwidth usage
Fast queries
Resilient against attacks
Long term storage of all the events
Explanation of Requirements:
Noone wants to miss out on events. If I am following someone I want to be sure that I am getting their events. There is no guarantee in current paradigm. If I am posting something I want it to go to everywhere. With lots of events shared between relays and eventually users, the interactivity and excitement will increase on nostr.
Right now the responses to queries are similar, coming from several or more relays, which makes nostr traffic to be very inefficient. Efficiency is important for especially mobile clients.
Current relays store the events for long term. Which adds load for most queries which look for recent events. Some relays can specialize in delivering recent events quickly and deleting after some time except maybe some kinds like 0 (metadata) and 3 (contacts).
Simply increasing the number of servers should deter some DoS attacks from even beginning.
Other relays can specialize for long term storage of everything else and not for speed.
Proposal:
To solve these I am proposing some specialization of relays. There will be like 32 read only servers which specialize in speed, fast queries, high bandwith, each temporarily storing a shard (1/32) of complete set of all events on nostr. The events can be sharded based on their last 5 bits. Storage can be mem only. If someone connects to all these 32 relays they will get all the ‘live’ events.
Secondly, the general purpose read/write servers which specialize in long term storage to make sure the events are preserved. They copy events to the read only servers (above). When a client wants to write, they can do to any of the general purpose servers. The events are then distributed to read only servers based on last bits of their event id. There may be hundreds of these servers.
Clients can connect to 32 read only plus some read/write servers in order to achieve everything.
We can have a copy of all shards to increase redundancy or resiliency against DoS attacks, or just to increase speed by having a lower ping.
Mobile clients need lower traffic. By connecting to 32 shards they are making sure they get unique events from each, thereby reducing the traffic. If mobile clients don't mind more querying traffic, # of shards (32) can be increased even more to 64, 128…
If # of shards is 64, last 6 bits of the events will be used for sharding purposes.