one of the slowest parts of receiving lots of events on mobile is actually json decoding. I wonder if it’s time to create an optimized binary transport for nostr notes. Could be a simple TLV format.
Discussion
That’s what I call an idea 👍
GM Will 🌞💜🧡🤙
I have no idea what you are saying but I know it will be awesome.
me reading this note:
Using the inbuilt Apple JSON support? Maybe we can try swap out to a faster external library?
The other major issue I see is websockets have no congestion management - it’s event by event, and a large event blocks all the smaller ones behind it.
This? “IkigaJSON is a really fast JSON parser. It performed ~4x faster than macOS/iOS Foundation in our tests when decoding a type from JSON.”
I tried to switch to this and it was much slower
Damn. Have you seen fiatjaf’s NSON proposal?
Yeah I don’t get it, I don’t know why it’s not just a new wire format. I have already replied on the PR about it.
i tried a branch where I built my own custom json decoder, but you still need to walk every byte in the string and unescape things. json decoding is not trivial, where a TLV format would be as simple as copying bytes directly.
I just realized this isn’t true, since the json encoded string is signed you would be forced to decode json strings regardless. Damn this probably isn’t worth it then.
Nm I’m dumb the decoded string is signed . I should have coffee before I start thinking of new data formats.
oof
Do you actually need to check the signature in this situation?
Right now relays are one of around five or so implementations. The issue is at some point when we end up with many more satellite (smaller) relays, malicious relays (or even software bugs) will start to appear.
Maybe trusted relays could have their events skip validation - however, at any moment validation could become really important to enforce client side.
You could run the validation async on the phone (e.g. first get data and then on demand verify signatures as the user scrolls through feed and maybe show a little checkmark when the verified and show red warning when not)
Yeah, on this note, first JSON decoding of multiple events can be trivially parallelized and second the rapidjson C++ lib is incredibly fast.
Finally, the biggest performance hit with this type of work is the heap allocator being used to allocate new memory for each object being decoded.
If you pre-allocate a bunch of space for the decoder output and use a format like rapidjson where the entire result uses one contiguous memory region it’ll be blazing fast without any need to make architectural changes to the protocol.
On second thought I do retract the “trivially parallelized” part. Parallelizing will add extra complexity and friction, so it’s not zero cost. But I stand by my other point of the heap allocator being the main culprit
Definitely
I don’t see why not 🤷♂️
Yes. And there is more of what should be done: https://damus.io/note12044688xrg67tnc2lkv5whwrvycy0y2rsrg6lgd8s8ses2pr57nsfv85x5
I need to automatically convert these links to note refs….
nostr:note12044688xrg67tnc2lkv5whwrvycy0y2rsrg6lgd8s8ses2pr57nsfv85x5
“Easy to implement” seems to be the right thing to do for me. The riping will follow with success.
I don’t think I understand what this man means but it talks about replacing json decoding with something else… if it’s j something it needs replacing, right? Thant’s how it works?
Would this mean that relays would need to support both formats? (falling back to json if the client doesn't support it)
Nope, relays could advertise that they support it or not. Would be completely optional.
But then you specifically request data in this firmat from the relay, so that you need no detection on client side, right?
It would be advertised in the relay metadata if it’s supported or not.
That part is clear, but how does the relay decide which format to send events in?
Great idear 👏
What library are you using?
Could you use a C++ library?
I've used this one, and its very fast.
json decoding is non trivial. The fact you even need a library is a downside.
I know. I just spent a year writing the JSON serialization for complex class trees for a product at work.
Binary has a large number of issues as well. Versioning, architecture differences (big vs. little Indian), and is not human readable for debugging. How are you going to handle additions to the protocol in a way that doesn't break existing clients, etc?
This is an optional optimization, you can ignore it. It is just advertised in relay metadata and if clients want to use it they can. Versioning is not an issue with TLV and we’re talking fixed formats here, notes and subscriptions don’t change much. If new fields are added to notes or filters then it would just be a new type.
have you considered using protocol buffers for the binary message?
why
That seems of dubious value. I think the 1+ minute gap between opening the app and having new events is the biggest deterrent to returning usage and is a better application of time & effort (I should help with this too)
😏
"Could be a simple TLV format" -- I would caution against developing custom formats. Many have tried to develop these formats and in almost all cases, it is far worse than the already standardized schemes such as avro/protobuf mainly cuz missed use cases or schema backward/forward compatibility issues. It is also hard for newbies to onboard on to these formats (although bitcoin chose custom formats but its history predated the standardized forms).
The only scheme/method that was different from previous standardized ones was flatbuffers by google-- they provided a way to decode without actually deserializing the entire payload at the expense of larger data payloads for big schemas.
If I were to do it, I'd just add an api to nostr that handles avro(for ex) and whether it is binary or json becomes merely a transport layer responsibility. It also standardizes and enforces at the schema level what fields are to be expected with what types.. etc,.
Flatbuffers and protobuf are meant for generic data formats that evolve over time. Nostr is a fixed data format, protobuf is overkill.
I agree TLV seems fine if you’re just encoding a stable schema. worst case is you’re incrementing types and clients have to add another case to their switch statement… which we already do with ‘kind’ today.
I'm curious what javascript/typescript library do you recommend to play around nostr apis?
I’m not much of a JS/TS guy but even I have been able to be successful with nostr-tools:
exactly
But new NIPs might introduce new tags. Should we consider this as well?
Does it really make a difference considering that signature verification takes much longer?
I don’t think most clients are verifying signature 😬 but I would love to be wrong about this. This is and should be happening at the relay level, but there is a client performance tradeoff doing it at the client level.
What I’d love to see if verification of every event isn’t possible client side: 1. Verification on profile metadata before someone zaps someone else 2. The ability to “verify” any post on demand if the contents are suspicious or really on any event containing an invoice
Damus doesn't verify sigs
CPU is bottleneck and not network?
I’m just talking about what I have seen to be the slowest parts of the codebase when profiling.
数据都是由三个域构成：标识域（Tag）+长度域（Length）+值域（Value），简称TLV格式。
用到数据编程时，我常直接用“｜”隔开各数据，一个split就变成数组了。
#Nostr 设计用JSON格式，我觉得主要目的是为了易读。从而能快速看到其签名的具体有内容，而不用复杂地转化查找。
lightning:cndx@nostrurl.com 🐇ᥬ[🐕]᭄🌿
Makes sense. Network latency is slowest link in the chain
Protocol Buffers? We moved from json over rest to protobuf over gRPC at work, been very happy with it so far.
JSON was a dumb idea...
#[0]
How about MessagePack? Faster to encode and decode than TLV, good for dynamic data structures apparently.
“TLV, JSON, and MessagePack are serialization formats without predefined schemas, enabling flexibility to represent a variety of data structures.
The protocol employing the format(s) establishes the rules necessary for correctly interpreting the encoded data.”
Never looked at MessagePack.
Last time I had to design a consensus critical message format I came to the unenviable conclusion that none of the existing ones were right. But that was for something different than nostr.
What’s the logic behind that conclusion? Would be nice to avoid reinventing the wheel.
I’m curious about what problems you encountered.
I do like the simplicity of TLV and how it doesn't inherently define specific data types. Lightning using it seems promising.
Minor benefits of MessagePack probably aren’t worth its complexity.
👏
#[2]
flatbuffers are nice, TLVs and flatbuffers are about the same. I would steal the null terminated string idea to make TLVs zero copy for strings.
Could be backwards compatible too, just switch from text messages to binary over the websocket
MessagePack is good, otherwise one of the ASN.1 encoding seems reasonable, especially given the association with cryptography (e.g. DER).
JSON is a also not friendly for binary data. It's very useful to have a binary format. Do you have a proposal in mind?
Here is the proposal addressing this issue: https://damus.io/note1wra64ux3w5fwxjd3y3v63pwxkmx64p57564zkjk6xj0a87k8keuqy07l34
nostr:note1wra64ux3w5fwxjd3y3v63pwxkmx64p57564zkjk6xj0a87k8keuqy07l34
I think that was a hard fork
Yes I saw he updated it to this later, my bad: https://github.com/renostr/nrps/blob/master/nrp-1.md