nostrdb stores notes in an optimized binary format, for 1 million notes its less than a gig. The operating system is responsible for LRU purging page caches so I don’t have to worry about it. You could have a hundreds of millions of notes mapped in 1TB of virtual memory and it would work fine.
Im using the on-disk db as my session cache in that sense, resident memory use is very minimal since data is only loaded by pages in parts of the db i need, and all that is managed by the OS.
Right now I only store encrypted notes in nostrdb, so I might need an in-memory cache for decrypted stuff. The in-memory format for compact binary notes is the same as on disk, so you get the memory savings there as well. I still use this for damus ios and it made things a lot faster and use less memory.
This is actually explicitly supported in the rust bindings, as you can see you can have an owned or db-owned note, where the db one is tied to the lifetime of a transaction. But in either case its just a pointer to memory or virtual memory
Simple apps should be able to handle decryption while scrolling just fine. So, I am not sure if anything more complicated than that is worth including in your lib. But more complicated UI components might trigger way too many pre-processing algorithms in parallel when scrolling and then pre-caching is the only viable answer. And storing this cache can be beneficial if the sum of all the pre-processing stuff to create those caches on the fly takes longer than loading them from the disk, which in many of our instances, already do.
Some time ago you mentioned that you were adding the .content parser to the db lib as well. That aligns with the cache needs here as well. Being able to pull already parsed .content from the db can provide significant gains in the rendering performance just because the CPU is left alone to do the other things the app might need.
Sure, but the question was if we can add more pre-processed attributes (key-value pairs) to the Event objects so that when it loads from the disk it's not just a raw event. Most of the processing Amethyst does is in the "getting an event ready for the UI" phase, like zap amount sums, reply counts filtered by the user's hidden words list, creating blurhash bitmaps, tables for zap comments per note etc. That's were 95% of the app's CPU usage goes.
Ah yeah that’s the same with damus, which is why I’m trying to put most of that upfront work into nostrdb. i feel a bit uncomfortable putting too much business logic into it, but the multithreaded ingester at least does content parsing, fulltext indexing, stat counting (zap amounts soon, likes, etc). Once it’s all ready the subscription is then updated for the UI.
Content is parsed into “blocks” which is also queryable and available for all notes. This was the slowest code for damus.
I want to put in all the complicated stuff that would be common in all the clients I build.
I wouldn't hardcode any specific business logic. I would just have a map
our metadata table achieves this now! you can add metadata of any type to nostrdb that is tied to a note id. currently powers our like/quote/reply counts
