nostr:nprofile1qqswa8vhnelpgx9f7arjhtuzmjtqs2sdgfgmw77tzu9xankf87kl7eqppemhxue69uhkummn9ekx7mp0qyfhwumn8ghj7am0wsh82arcduhx7mn99uq3kamnwvaz7tmgv9mx2m3wv93kx6t0d3ujuum0vd5kzmp0gy8dgf maintains a repo with Nginx reverse proxy included. Also have a look at his repo, if it fits your needs.
Running haven as a personal relay is the #Nostr equivalent of running your own #Bitcoin node. nostr:nprofile1qqsq4hm8gawvch9y2m7nqghydaw4ym4s4a3gf0u9f9xqm4uy0ul9qvcpremhxue69uhkstnrdajxjmn8v9ex2mnp9e6x7up0da6hgcn00qq3vamnwvaz7tmwdaehgu3dwfjkccte9eshqup0qyghwumn8ghj7mn0wd68ytnhd9hx2tcht4zgq maintains a Docker wrapper, thanks for that!
Discussion
Hey Ben, thanks for the continuous support! :) My repo is long overdue for an update by the way. I’ve been working on it but couldn’t get the work finished up over the weekend. Hopefully, it’ll be ready today. I’m planning to release an update with the latest upstream version of Haven, along with some extras, including Nginx and LMDB optimisations, keepalive enabled by default and enhanced support for macOS, Windows, and Linux on Arm.
Between:
1. nostr:nprofile1qqsrhuxx8l9ex335q7he0f09aej04zpazpl0ne2cgukyawd24mayt8gprfmhxue69uhhq7tjv9kkjepwve5kzar2v9nzucm0d5hszxmhwden5te0wfjkccte9emk2um5v4exucn5vvhxxmmd9uq3xamnwvaz7tmhda6zuat50phjummwv5hsx7c9z9's changes to EventStore and Khatru fixing a bug that caused outdated replaceable and addressable events to linger indefinitely (plus introducing optimised COUNT support)
2. nostr:nprofile1qqsw9n8heusyq0el9f99tveg7r0rhcu9tznatuekxt764m78ymqu36cpz4mhxue69uhkvun9deejuat50phjummwv5hsz8rhwden5te0wfjkccte9e3xjarrda5kuurpwf4jucm0d5hsz9thwden5te0wfjkccte9e6hg7r09ehkuef0avzrjf's upstream updates (by the way utxo, I’ll have a small PR upstream soon to enable NIP-40 / Expiration Timestamp support as well)
3. A few of my own additions already mentioned above
IO and CPU usage for my local Haven container has dropped by over 40%.
Stay tuned!
Did I fix that bug? I don't even know.
All I know is that you should all be using the ReplaceEvent hook.
Yeah, you did 🤣. The outdated replaceable and addressable events (including the infamous outdated 10002 lists that have been bothering me forever) are now being properly deleted. I no longer need to run my crappy maintenance scripts to delete outdated events! If it wasn't intended then I think this was a "happy side effect" of your changes to fix pathological duplicates. Likely the changes to the Publish method in this commit, but I'm not 100% sure: https://github.com/fiatjaf/eventstore/commit/295a7510337c39e79d5c8154d22d1fb86438098b.
And you’re totally right. I'm already working on adding ReplaceEvent handlers to Haven. I managed to break my own dev branch this weekend while updating Khatru and EventStore to the latest version (likely due to conflicts with other libraries). But as soon as I have a stable version, I'll submit a PR to utxo's repo upstream.
OK, well I thought there was a chance I had fixed it but wasn't 100% sure.
Thank you for monitoring all of this and ensuring things continue to mostly work.
nostr:npub1utx00neqgqln72j22kej3ux7803c2k986henvvha4thuwfkper4s7r50e8. You can find the PR for this here: https://github.com/bitvora/haven/pull/61. Tested on Windows, macOS, and Fedora (both x86 and ARM).
I decided against implementing the compress flag because outdated events were also being retained in BadgerDB. As saintly as nostr:npub180cvv07tjdrrgpa0j7j7tmnyl2yr6yr7l8j4s3evf6u64th6gkwsyjh6w6 is, he has only implemented compression logic for LMDB. I'm too lazy to add compress logic for BadgerDB in event store, and it doesn't make sense to implement the compress flag otherwise.
In practice, some old replaceable and addressable events will naturally be deleted as users write new versions. If users find themselves with millions of outdated events stored in the database, as I did, they can always nuke the database and reimport. The different in performance is impressive.
Please let me know once you’ve merged this (or if you need any additional effort on the PR) so I can cut a new Podman/Docker version.
What compress logic is that? You mean the migration step on LMDB that reindexes all the events? I forgot to do that on Badger because I wanted to see if it would work first.
Oh, now I see what you mean. I don't think that compact stuff is relevant here. The thing is that both on Badger and LMDB whenever we got a replaceable event stored twice (apparently because of race conditions) that would break the indexes in some way I forgot, such that it became impossible to delete the older version forever because it would be unreachable. The only way to fix this is to do a full rescan of the database and rebuild all the indexes, which is what LMDB does in its latest migration (well, all the other migrations I deleted because this big rescan makes them unnecessary).
Badger needs that same thing, but I forgot to do it. In any case you can do it manually by copying then nuking the database and then reimporting the events from the old database to the new empty one. I should still write the reindexing migration though.
Hi fiatjaf, yes, that’s the one. At 2:00 am “compact” somehow became “compress” in my head. But you’re right, other than making certain “immortal” events deletable on LMDB as per your original intent above, for the end user, it’s essentially a major database-wide deduplication (which is exactly what I was looking for with BadgerDB).
You’re also right that nuking the database and reimporting old notes has the same effect. This is what I’m suggesting for Haven users for now. Unfortunately, Haven can’t import its own backups (yet), but users can always reimport some of their old notes from other relays or temporarily use a second instance of Haven to do this. I was just considering a Haven-specific --compact flag for completeness (e.g., so users don’t lose private notes that aren’t currently reimported) and to save them the trouble of doing this manually.
Either way, awesome work. Many thanks! Haven is absolutely flying with the new Khatru engine. I even tested this with an old LMDB database backup I keep around for testing purposes. Compacting the database cleared out over 2 million duplicsted events from a database containing only around 1k short notes. It’s impressive how clients continuously spam lists, sets, etc. Now I know there was much more to it than just the Amethyst kind 10002 write loop bug.
Replaceable events should only be used for things that are written sporadically. There some shady stuff being done out there with these, I think it's wise to only allow some explicit whitelisted kinds that we know aren't spammy.
💯 To be fair, I’m encountering broken client behaviour or client/relay incompatibilities that result in spammy activity with otherwise "legit" events more often than actual malicious code or directed attacks. But you’re right, we should be doing something about it. Whitelisting specific events is a good start. Maybe I’ll build a Citrine-style dashboard for Haven so users can at least get a sense of what they’re storing in their relays. From there, we could add functionality for deleting individual events, deleting all events of a certain kind or even blocking them entirely.
For now, though, ReplaceEvents are doing a great job of preventing unnecessary database bloat. Again, many thanks.
I'm not talking about malicious stuff, but things like Amethyst draft events that rewrite the same addressable a thousillion times (I'm not sure this actually exists but I've heard it is a thing).
I’ll have a look deeper. I haven’t paid much attention to Haven’s private relay since Inbox and Outbox are always the ones on fire, but apparently, I only have three kind 31234 events (Amethyst-style drafts) across all my relays. Draft events are certainly high-frequency, but as far as I can see, they aren’t bloating the database.
List and set events, on the other hand, have been the bane of my existence. That, along with the fact that Amethyst still doesn’t send the right events to the correct types of relays, remain my top two unsolved tech problems on Nostr. Vitor mentioned he was working on it, but it’s a non-trivial change given how much functionality has been built on top of the classical general relay model. Fingers crossed, both clients and relays will see some improvements this year. I’d prefer these two fixes over any new features.
I know what I'll deploy to my cluster tomorrow.