Nostr Web Client

Does #lmdb make sense for an embedded db? I think #sqlite would be better. Querying would be difficult otherwise.

It is a trade-off. SQLite like any RDBMS with SQL lets you whip up new queries without programming anything new. I lose that. I need to create my own indexes, and write my own code to compute what an SQL query would do automatically, which might be hash join or something. But so far most access has been pretty simple and I haven't needed to code any tricky query logic yet. In the worst case I could run an EXPLAIN command on a database and see what it would do, and then code that.

LMDB gives you such fast access and iteration that you can consider it as being in memory already. So with SQLite I was copying into memory caches and then trying to keep them in sync - I'm ripping that stuff out. I was also doing all the database calls on separate threads so it wouldn't stall the UI - I'm just doing them direct now. With SQLite I had to do object-relational-mapping in many cases. With LMDB I just seralize the object into place (although strictly speaking I could have done that with SQLite and stored objects as BLOBs but then I would lose much of the SQL ability).

I think that if your data needs to be shared on a corporate network with many different applications and users accessing it, and you need to support any possible query being thought up in the future, then RBDMS is the right solution. It is much less clear when the data is only accessed by one application, and I'm leaning towards LMDB at the moment.

Reply to this note

Please Login to reply.

Discussion

Mike Dilger ☑️ 2y ago

So here is a real-world example. I used to do this: Notice I get to use SQL to update all the relays in the table.

pub async fn clear_all_relay_list_usage_bits() -> Result<(), Error> {

// Keep only bits which are NOT part of relay lists

let sql = format!(

"UPDATE relay SET usage_bits = usage_bits & {}",

!(Self::INBOX | Self::OUTBOX)

);

spawn_blocking(move || {

let db = GLOBALS.db.blocking_lock();

let mut stmt = db.prepare(&sql)?;

rtry!(stmt.execute(()));

Ok::<(), Error>(())

})

.await??;

Ok(())

}

Now I have it like this

pub fn clear_all_relay_list_usage_bits() -> Result<(), Error> {

GLOBALS.storage.modify_all_relays(|relay| {

relay.usage_bits &= Self::INBOX | Self::OUTBOX;

})

}

pub fn modify_all_relays(&self, mut modify: M) -> Result<(), Error>

where

M: FnMut(&mut DbRelay),

{

let mut txn = self.env.begin_rw_txn()?;

let mut cursor = txn.open_rw_cursor(self.relays)?;

let mut iter = cursor.iter_start();

while let Some(result) = iter.next() {

match result {

Err(e) => return Err(e.into()),

Ok((key, val)) => {

let mut dbrelay: DbRelay = serde_json::from_slice(val)?;

modify(&mut dbrelay);

let bytes = serde_json::to_vec(&dbrelay)?;

cursor.put(&key, &bytes, WriteFlags::empty())?;

}

drop(cursor);

txn.commit()?;

Ok(())

}

(the serde_json serialization is not ideal, but easier since my relay objects have arbitrary JSON content due to containing the NIP-11)

jb55 2y ago

Are you using mozilla/lmdb-rs ?

Mike Dilger ☑️ 2y ago

yes, lmdb-rkv at https://github.com/mozilla/lmdb-rs

jb55 2y ago

Was looking at that earlier today but was spooked by the issues mentioning iterator crashes 👀

Mike Dilger ☑️ 2y ago

Seems they've been around at least four years on this repo, and probably longer on the repo this was derived from. If you don't try to drop a cursor and keep using the iterator derived from it, you should be okay. Why they can't have rust enforce that ¯\_(ツ)_/¯