Global Feed Post Login
Replying to Avatar Vitor Pamplona

Devs, has anybody built a memory-efficient Event Hint indexer?

Basically, I want a separate in memory cache where I store all event IDs the app has seen and all relays they can be found on, merging receipts "seen on" with the hints in all tags.

The idea is to decouple event loading from the information in a particular event. Instead of using just the hint provided by the event, the app should check the hints for all other events received.

For instance, we could do a Bloom filter for each relay to avoid storing event ids in memory (and in disk). If I need the relays for a given event ID, I just check the presence in all Bloom filters from the lowest to the highest ping.

We could use the same for pubkey hints and address hints as well.

My only concern is the computational expense of doing multiple hashing rounds for each add and contains operation for every hint in each event the app receives. 🤔

Avatar
Girino Vey! 10mo ago

you can probably store just part of the event id, since its a hash, and collisions will still be pretty rare. You can adjust how much of the ids you keep based on the amount of available memory.

Reply to this note

Please Login to reply.

Discussion

No replies yet.