Nostr Web Client

i've been trying to have some kind of sensible "database drop" functionality in my project for testing purposes and i initially had this funny iteration that went through and deleted everything

this broke two funny records that made the migration thing bomb out and refuse to start

so then i made a thing that iterated and left those two records... but even though i close the db, somehow those delete key operations didn't actually work

and then, finally, i find there is a bulk drop operation, that lets you give a set of key prefixes to drop the damn records

et voila

finally!

this is for the #golang badger key value store... i'm starting to get a bit handy with this thing now, it's a really nice, really fast db with some very neat capabilities for creating indexes (because they can be key-only, and the design separates keys and values to mitigate write amplification in the log-structured (LSM?) key value store

nobody else in key value store dev has bothered to add this feature to any database key value store for any other language... not one... but screw them, because dgraph uses it as it makes their graph database that much better, and it is a really solid excuse for me to say "learn #golang so you can use a decent database with modern features"

just do it... learn golang, you will never regret it, the FUD around go is bizarre

#devstr #deardiary #badgerdb

Reply to this note

Please Login to reply.

Discussion

Setor 1y ago

How does badgerdb compare to leveldb?

ᴛʜᴇ ᴅᴇᴀᴛʜ ᴏꜰ ᴍʟᴇᴋᴜ 1y ago

it has a separate key log which means you can iterate keys independently and add/change them without impacting the value log, which is where you get "write amplification" which has to do with the fact that the value tables are much bigger and cost more to restructure and append to

this makes it much more viable to build powerful search indexes by simply using the keys, which is exactly how the badger eventstore that fiatjaf wrote works to implement filter searches, there is like 8 different kinds of keys that let you match fast to authors, timestamps, and so forth, as well as a scheme to add a key for each tag in an event

implementing fast range searches with badger would be much easier to do and not have problems with large data sets due to this feature

ᴛʜᴇ ᴅᴇᴀᴛʜ ᴏꜰ ᴍʟᴇᴋᴜ 1y ago

i was avoiding paying attention to this part of the codebase from the eventstore/badger implementation but right now currently cleaning the hell out of it and restructuring it a little... in several places it needlessly trims down 64 bit values to 32 bit, which means that the database has maybe got a 2 year lifespan before the primary record keys overflow, and the timestamps... also shortened to 32 bit which means they also overflow circa 2038

not that these are big concerns right at this minute, for a high time preference programmer but for a serious, long-term codebase this is exactly the sort of thing that cannot be allowed to continue

by the time these numbers run out, at the 64 bit size, the potential size of storage capabilities is far beyond what currently will blow up in 2 years time with a mini y2k bug