Two weeks of hardcore performance work on Quartz, but now we can import 3GB containing 2.1 million contact list events in 9.4 minutes, *single-threaded* on a Pixel 9, using just a cool, calm, and collected 15MB (yes, MB) of RAM.

That's up to 10,000 events per second per CPU for lighter follow lists and 3,000 events per second on heavier ones.

By import, I mean receive, unzip, parse, string intern, verify, save, and index (SQLite) with individual transactions on write-ahead logging to guarantee DB consistency and match real use cases.

JSON parsing w/ string interning is now pretty much capped by IO and interning at 120,000 events per second per CPU or ~50MB/s per CPU.

JSON Parsing + interning + verifying runs at ~20,000 events per second per CPU or ~30MB/s per CPU.

None of these use SIMD instructions, which are still poorly supported on mobile CPUs.

Reply to this note

Please Login to reply.

Discussion

What is Quartz?

Wow, well done!

Impressive!

Unreal. Great work

I don’t want to have to spend 10 minutes waiting.

Wait, what is Quartz?

Kotlin library for nostr

Also. Don't worry we are not downloading 3GB of events on Amethyst yet :)

the question is, how much extra cost is coming from indexing every single word (and URL) like this, that's a full second random iteration. i severely doubt that you are gonna do better than ZSTD at even 5 for space and speed.

on the other hand, you have a fulltext index there now, iirc you said it works with an inverted index?

Interning for now is only in memory. It significantly reduces the amount of duplicated Strings in memory for sure. Especially when loading p or e tag heavy kinds like follow lists

definitely is gonna help with follow/mute lists that's for sure.