Nostr Web Client

How do you measure if this is worth doing? I did something similar on my recent DB adventures, but I realized that the bulk of the size on Nostr comes from the tag array and content, not the other fields. I ended up referring to the hex representation because it is much easier to work with and it doesn't really add much to the db (<1% in my tests).

Danny, the cyber guy 5mo ago

personally I think this could be great as an over-the-wire format, rather than storage, since as you're pointing out it makes no difference in storage.

the biggest advantage is that it's super simple to encode and decode (simpler for your cpu than json, in fact!).

sending 1000s of events from a relay could save hundreds kilobytes of data (which is great for areas with bad internet, or phone connections), and plenty of cpu cycles on both the client and relay (i.e. better for older devices, and cheaper servers)!

Reply to this note

Please Login to reply.

Discussion

jb55 5mo ago

I think nostrdb format is better for storage since its flatbuffer style and doesn't require serialization in or out of the DB.

this is definitely better for over-the-wire, but i'm confused when ya'll are saying its not good for storage, since it will detect and encode any lower-cased hex string in the note as bytes, which is great for reducing storage sizes, many times its 50% storage reduciton.

but yes its also crazy cpu efficient.

Vitor Pamplona 5mo ago

Nah.. the wire is gzip compressed with almost every relay these days. The gain for comms is minimum.

jb55 5mo ago

you can skip gzip and json decode using this format, and its smaller. thats lots of cpu savings.

Vitor Pamplona 5mo ago

I don't think it is smaller. Did you test it? A 300 people contact list goes from 23kb of minimized json to 12kb as gzip. And those have the lowest sparcity. Any text note compresses like crazy.

jb55 5mo ago

yes I tested it:

36746

39742

zstd beats it:

35530

but you still have to un-zstd and json decode..

jb55 5mo ago

i'll try to see how much faster it is with benchmarks

Vitor Pamplona 5mo ago

Yeah, those numbers make sense. I don't know, I feel like beating a dead horse when we try to beat compression algorithms.

jb55 5mo ago

another nice thing is that this format ensures the id is at the start, meaning its very easy to reject parsing/verifying the entire thing when when checking if you already have it:

for field in packed_note {

// first field peeked when looking at the bytes

if let ParsedField::Id(id) = field {

if cache.contains(id) {

break;

}

I had to really hack this in my json parser on nostrdb to get this working

jb55 5mo ago

its not bad nostr:note1t3cnac2fqr4wcl6a8eve63q3577e77p8rgrgpak8mn77py4dhnfqxwrnyc