#orly #devstr #progressreport
after much lying from claude and gpt-5 i eventually got it to finally build the benchmark comparison so it works, and here is the result:
```
================================================================
BENCHMARK COMPARISON TABLE
================================================================
Relay Status Peak Tput/s Avg Latency Success Rate
---- ------ ----------- ----------- ------------
next-orly OK 17901.30 433.058µs 100.0%
khatru-sqlite OK 14291.70 545.724µs 100.0%
khatru-badger OK 16351.11 474.016µs 100.0%
relayer-basic OK 16522.60 466.066µs 100.0%
strfry OK 15346.12 506.51µs 100.0%
nostr-rs-relay OK 15199.95 508.699µs 100.0%
```
ORLY is number one for latency and throughput. second fastest is relayer (which is using postgresql database) of course the slowest is khatru using sqlite3 database, but it's not really that far behind strfry and nostr-rs-relay
here it is in more detail, for bulk query only test, ORLY is actually the slowest, but keep in mind these tests are quite short and the database is not compacted for optimal seek latency. it still however performs just as fast as purely storing new events as if it is being queried at the same time, so that's probably partly where the tradeoff is. it should be pointed out also that ORLY's tables are probably going to be bulkier than most others because it is designed so that it can fetch intermediary tables that contain ID, pubkey and timestamp so it can get them in correct order immediately and then stream them to the socket.
there will now be a metric for me to evaluate optimizations against. i probably will extend the benchmark to run a longer evaluation with bouts of quiescence that will ensure all of the databases will get a chance to compact their tables.
committing this now, also, any requests for other relays to add to the test, or different versions of ones that are in there to add to it.
i'm sure i can make it even faster yet, but this is a great result, slightly slow on pure read but 6% faster than khatru and relayer, and 13% faster than strfry and nostr-rs-relay
gonna focus my first efforts at speeding it up with query performance once i know how much of this lag is from fragmentation of tables.