Looks like that worked. so it can shrink the database but it cant grow it?

Reply to this note

Please Login to reply.

Discussion

Yes, it can. The short-term "right" fix here is to start small and grow the MapSize on demand, which, again, may or may not preallocate space fot the DB file depending on the OS, filesystem, and architecture. For example, on Windows and macOS ARM, the initial MapSize has to be smaller than the available disk space, but once it has started (despite file managers displaying a 273GB file) it’s not actually using that much disk space.

I considered implementing dynamic growth in Khatru’s EventStore, but it’s a reasonably complex, non backwards-compatible change. And when I discussed it with Fiatjaf, it seemed there wasn’t much appetite for it.

There are other LMDB forks/wrappers that handle this more gracefully, so Fiatjaf is probably right. The best long-term approach is to migrate db libraries or db technology altogether. That would likely also help with some other Khatru issues around migration, transaction size, etc. But that’s a separate conversation.

The static MapSize parameter is good enough for Haven’s needs, and I assume for most other Khatru relays as well. Relays with the pretension (or real need) to scale on demand, potentially to petabytes in an enterprise DB cluster won’t be well served by EventStore, and to be honest, not even by Khatru in its current form.

Still, that would be a good problem to have, and right in my area of expertise (Distributed Data-Intensive Systems) when we get there. Semisol is already building something along those lines, so he’ll likely be the one pioneering this kind of thing on Nostr.