Let me know what you think https://github.com/mleku/orly/pull/10
Discussion
looks almost perfect.
relayer has an error in the "basic" relay interface implementation, i have fixed it and made a PR but for now i'm pointing it at my fork so it compiles.
i'm updating it for the v0.7.0 branch now, and will bump t, v0.8.0 to signify the dockerization
ok, i've fixed all the builds so they work, but the actual benchmarks, it doesn't seem to be running the relays for the benchmark to connect to.
as far as i can tell, the relays are not being started up
I'm going to revise it as a simple docker that just does all the tests in a single container.
It builds everything fine now, so I think I just write a shell script to run each relay, bench it, stop the relay and continue until all done, and then write the report to the terminal.
I can't figure out how you got it working when the relayer basic doesn't build due to a change of interface methods. The script of the strfry copies a file to itself, causing the dockerfile to abort. I will switch khatru to run the badger also, because I can't wrap my head around networking. I do know for sure that the badger database indexes fiatjaf made do a lot of event unmarshaling due to an incomplete set of indexes.
I assume an LLM coding agent did this. They are good at this kind of dumb, repetitive stuff. But it seems to me like surely it hallucinated the results. Anyhow. I will get it working today.
The docker set up still isn’t perfect, I still had to do manual work to get them all running but the numbers are from actual runs of all 4 relays to compare.
If you want I can spend more time getting it set up but things like strfry were a pain and I had to do that manually. The only one that worked fine to run the benchmark was orly and khatru.
i have spent much of my morning writing a script that installs all the dependencies, go and rust, and i added nostr-relay-rs. i'd think it would be trivial to turn it into a dockerfile.
anyhow, i'm gonna see if i can't get the benches to run without containers, at least on ubuntu 24 this script does everything, takes about 20 minutes, half of that being for strfry and nostr-relay-rs.
just gonna understand how this benchmark tool is invoked, and learn how to grab the pid of a background-launched process to kill it after the test, other than that should just be a matter of following along with how the benchmark tool is run in the existing configuration. should be simple to make it all into a single container.
i dunno why it didn't want to run for me. i can only speculate something to do with docker. used docker compose v2 and everything.
in any case, the script is in there now, at the HEAD of the main branch. the most important thing is the benchmarking program you built, that's the heart of the thing. the build stuff is just tedious crap and reading stupid bad documents :)
ok, apologies on all the back and forth, it is a pain running some of these relays (I spent a while getting them all running to produce the new benchmark comparison results), but I'm working on a branch I'll push up with some fixes, hopefully helpful.
for me, obviously, part of it is going to be about being vindicated about my methods and choice of language. raw throughput, concurrently, will show a lot but the profiling and latency measures are also very important.
i refactored the database to perform the queries entirely in one thread, and made the transactions smaller so that they should splice together concurrently well. this is one thing that i want to get right, i will probably be doing more profiling on the concurrent random querying to find where the bottlenecks are. to be honest i feel like there probably is some, and probably because of some relatively new code that was almost completely written by an LLM coding agent. i already noticed a lot of random and stupid stuff that was in code it generated so i'm sure there is some low hanging fruit there too.
it's my opinion that the less latency there is in request processing, the broader the use cases. stuff like ContextVM and possible realtime collaboration protocols need this to be optimal.
i had really bad experiences working with the khatru codebase when i forked it 18 months ago. there was way way too much goroutines competing for resources and it literally ended the possibility of any further work through the sponsor who paid for that work. most of the code i wrote has ended up in ORLY tho. i wrote a lot of fast encoders and more recently experimented with better database index designs. but it's possible that there is contention in there or maybe there could be more concurrency. anyhow.
yeah, i'm gonna work on the script to run the tests, anyhow right now. i would like to see that working before i finish my day today.
Is your script using the rust relay on main or another branch? Unless you’re still working on it, I’d like to try it out as I haven’t ran the benchmark on the rust relay yet.
be my guest :) i picked the one with the most stars on github, i think it's the most common one deployed also.