> I've built a custom in-memory note representation...
That is quite the trick, much easier to code in C than in rust. I love this idea.
This will be faster than anything I code for some time. I'm serializing/deserializing and even though I'm using a fast library (speedy) it still requires an allocation and a copy at the minimum.
Thanks for pushing me in this direction, i was debating weather to go the simple route with sqlite, but since nostr is so restricted in its data format i feel like a custom DB with a dedicated mapping would work well.
How much faster will it be than sqlite? And how much value will it add to your most important projects? It's tempting to do cool stuff but ultimately it always takes (me at least) much more time and effort than anticipated
I think about 20-100 times faster. Not for the whole program, just for the data access part.
People love the idea of 'zero copy' but 'zero allocation' is where the real gains are. Memory allocators are complex and sometimes slowish things.
I've imagined a library that manages normally cloned data (mostly strings) by owning them in some global singleton object and handing out Arc references to them (actually has to be offsets, not actual references, if you want to persist and restart your code). Mainly so that I don't have to do .clone() and .to_owned() all the fricking time feeling like I'm slowing down the code, and so I can impl Copy on all the structs and as a side effect you could access them directly in LMDB without serialization. That system also has to persist to LMDB.
That's... impressively faster!
