Oh no, I missed this one, thanks for linking! This is a really good write-up and there are a lot of similarities with what I'm designing for strfry proxy. Like some of the comments, I'm not sure if partitioning on ID will be optimal in all cases. I can also imagine variations on pubkey and created_at.

Some variant of consistent hashing will also be necessary I think, for failure recovery, rebalancing, changing the number of backend relays, etc.

https://github.com/nostr-protocol/nips/pull/1494

I just tagged strfry 1.0.0. Here are some of the highlights:

* negentropy protocol 1: This is the result of a lot of R&D on different syncing protocols, trying to find the best fit for nostr. I'm pretty excited about the result. Negentropy sync has now been allocated NIP 77.

* Better error messages for users and operators.

* Docs have been updated and refreshed.

* Lots of optimisations: Better CPU/memory usage, smaller DBs.

Export/import has been sped up a lot: 10x faster or more. This should help reduce the pain of DB upgrades (which is required for this release). Instructions on upgrading are available here:

https://github.com/hoytech/strfry?tab=readme-ov-file#db-upgrade

Thanks to everyone who has helped develop/debug/test strfry over the past 2 years, and for all the kind words and encouragement. The nostr community rocks!

We've got a few things in the pipeline for strfry:

* strfry proxy: This will be a new feature for the router that enables intelligent reverse proxying for the nostr protocol. This will help scale up mega-sized relays by allowing the storage and processing workload to be split across multiple independent machines. Various partitioning schemes will be supported depending on performance and redundancy requirements. The front-end router instances will perform multiple concurrent nostr queries to the backend relays, and merge their results into a single stream for the original client.

* As well as scaling up, reverse proxying can also help scale down. By dynamically incorporating relay list settings (NIP-65), nostr queries can be satisfied by proxying requests to external relays on behalf of a client and merging the results together along with any matching cached local events. Negentropy will be used where possible to avoid wasting bandwidth on duplicate events.

* Archival mode: Currently strfry stores all events fully indexed in its main DB, along with their full JSON representations (optionally zstd dictionary compressed). For old events that are queried infrequently, space usage can be reduced considerably. As well as deindexing, we are planning on taking advantage of columnar storage, aggregation of reaction events, and other tricks. This will play nicely with strfry proxy, and events can gradually migrate to the archival relays.

* Last but not least, our website https://oddbean.com is going to get some love. Custom algorithms, search, bugfixes, better relay coverage, and more!

It waits forever, so plugins should ensure they do any timing out on their side. I have a debugging script that can be used to detect a stalled/missing response. More info in this issue: https://github.com/hoytech/strfry/issues/112

Currently strfry always waits until it gets a response about the previous event before sending another request, so the order is not applicable. Eventually I might make the implementation asynchronous in which case yes, mixed order would be acceptable. This is why the event ID must be included in a plugin reply even though technically it's not needed today. I clarified this in the docs: https://github.com/hoytech/strfry/commit/d2d8dc7572e63d74caf1810673981add47c9f328

Thanks, I'll give it a try!

I should mention that the existing tests and the differential fuzzing described in the strfry README pass on the 1.0 series, naturally.

Glad to hear! Thanks for letting me know, I appreciate it!

Heh yeah that is a pretty cool use-case actually.

But this feature was removed from NIP-01 and strfry tries to follow the spec as close as possible, so take it up with the nostr CEO. ;)

Thanks!

This change shouldn't make much difference in import's behaviour. Because import pretty much constantly has a writer lock, relay *write* operations will be blocked/delayed (and may timeout), but *read* operations should continue as normal.

We could make import release its lock for a few milliseconds periodically so that write operations can go through. I'll look into doing that.

I just tagged 2 strfry releases: 0.9.7 and 1.0.0-beta1

1.0.0-beta1 is a candidate release of strfry 1.0.0 -- help is needed testing!

The internal strfry DB version has been increased to 3, which means that you will need to rebuild your DBs to use this new version.

0.9.7 has some bugfixes and changes that accumulated in master, and has a "strfry export --fried" feature that can be used to create DB exports that can be rapidly imported by 1.0.0 series releases.

The full changelogs are available here: https://github.com/hoytech/strfry/blob/1f34794945cf380035822278c46070a9923129f3/CHANGES

Thank you to everyone who contributed and helps testing! If you need help or run into any issues, reply on nostr or stop by our telegram channel.

Yes, I saw -- it means a lot, especially coming from jb55. Thanks!

Exporting and importing events into a new strfry instance (which you need to do when the DB version changes) takes too long. Here's a feature I just added that speeds this up a lot:

https://github.com/hoytech/strfry/blob/next/docs/fried.md

Going forward, there is a release-0.9 branch. I'm going to tag one more release on that branch soon (after back-porting a couple fixes). It will have strfry export --fried but not import. I'm planning on this being the last release of the 0.9 series.

I'm working on a 1.0 release in the next branch. I just did a big refactor of the DB format that I've wanted to do for some time. I also removed prefix matching on id/pubkey (this was removed from NIP-01) and fixed a bunch of bugs. This release will also have the latest negentropy protocol version and BTree code.

Hi all! I have just pushed a major update to the negentropy project. It implements protocol version 1 (previous was version 0).

The protocol has been simplified: ID size is no longer configurable, there are no more continuation ranges, and the output can be constructed in the same input-scan pass.

There is a new fingerprint function, based on an incremental hash. This allows fingerprints to be pre-computed and stored in a tree structure. The C++ implementation includes a B+Tree implementation that allows fingerprints to be computed without collecting IDs from the entire DB.

I have written a comprehensive article that goes over the theory of RBSR and the negentropy implementation:

https://logperiodic.com/rbsr.html

Comments are appreciated!

Finally, I have integrated the new version of negentropy and the B+Tree with strfry. It's in a development branch and not quite ready for production, but my testing indicates this will be a massive improvement for full-DB syncs, especially on relays with very large DBs. In-sync or nearly in-sync relays should sync almost instantly with negligible resource usage. Relays will also use the B+Tree for filters that contain only until and/or since, meaning that date-range full-DB syncs are also efficient. Syncing arbitrary filters works as before (but I have begun work to make these more efficient as well).

Unfortunately, this is a breaking change for the negentropy protocol (this really should be the last one!) and will also require a new DB version.

I'm going to take this opportunity to make a few more breaking DB changes, and plan to release strfry 1.0 after a beta testing period.

This is a bit challenging, but I will look into it. Another possibility is we could try to optimise the import flow. Right now it's single-threaded and there are a bunch of other things that could be tuned also.

Hah, nice rant! Yeah sorry, my code is a bit unconventional. I'm happy to offer you a full refund. :)

FWIW there are 2 `Dockerfile`s in the repo: One for alpine and one for Ubuntu. If you get it working on another platform, I'd be happy to merge a pull request.

Thanks for letting me know, glad you enjoyed it!

FWIW I agree with you. Right now the front page is really boring. I have some ideas and I'm working on it!

It's based on kind 7 "reactions", roughly. It gets adjusted by the community's algorithm, and not everyone's reactions count (otherwise it would be trivially gameable by making a bunch of random pubkeys). I'll have more info on this soon, once community algorithms are ready.

Yes, it's actually embedded into strfry executable as an "app". WIP code is in the web branch of the strfry repo: https://github.com/hoytech/strfry/tree/web/src/apps/web

Yeah it's not great right now, sorry. Work in progress, stay tuned :)

Thanks all! I wasn't quite ready to launch this thing, but I really appreciate the feedback, cheers!

I was having some stability issues which was why the site was down for a while earlier today. It should be fixed now.

I just landed a new feature in strfry: "strfry router"

Docs: https://github.com/hoytech/strfry/blob/master/docs/router.md

I kind of think of this like "nginx for nostr". It lets you manage many up/down/both streams to remote relays using a single process and config file, and you can edit that config file without interrupting existing streams (mostly).

You can also use nostr filters and/or plugins to filter the traffic in both up and down directions, independently.

Hey all! I'm going to be doing a talk on nostr and the architecture of the strfry relay. It's next week at the CppNorth conference in Toronto, Canada:

https://cppnorth.ca/speaker-doug-hoyte.html

If anyone's around, it'll be a pretty fun conference! You can enter NOSTR in the checkout to get a discount on the ticket.

Hey all! Here's a new nostr relay implementation I'm working on: https://github.com/hoytech/strfry

It's still in beta/dev, but pretty close to production ready. Most interesting feature is a merkle-tree based set reconciliation protocol for syncing messages between servers.