good luck having fun getting agreement on binary codecs
you have all the CBOR fans, and the Protobuf fans and the MsgPack fans and then there is retarded custom codecs like the one i made for database and the one fiatjaf made for database but at this point these only encode events
in principle a good idea but the only way you are gonna get this to work is getting several clients to adopt it and at least one relay that makes this available, and probably you still have to stick with websockets
CBOR
Gray beard approved.
you see, there is a problem, i don't like cbor after dealing with it in my work with bluesky
so why do you love it so much? why do you prefer it compared to msgpack or protobuf or flatbuffers or capnproto?
keep in mind this needs to connect together apps written in go, rust, javascript, java, c, c++ and maybe at least reasonable support for python and c#
from what i've seen, and it's been a while since i've looked that close, protobuf was the most well and fully-supported across all of these languages
personally, i would want to use flatbuffers, and currently the Go version of this protocol encoding is shitty, but i could imagine myself reworking it in a month or two into something that is really natural to use with Go, because i have already in fact written an on-demand protocol buffer encoder, twice, in fact, simple TLV thing that was based on ad-hoc dynamic interface array message descriptors
CBOR is RFC standardized, protobuf is Google's personal pet project and with their schemas usually also too complicated.
🎯
as someone who works with interfaces and not variants i don't like any protocol that doesn't have static typing and i honestly don't see the point of complex rpc specification compilers when it's just a friggin function call
about 10% of cases, partial failures can happen that should really return an error and a partial value (eg, a result being an array of fields from an array of inputs) but nope, can't do that with typical C++/Rust error handling idiom
this forces you to design APIs as single shot rather than batched which is a pretty rigid and performance limiting way to do things, especially when it is easy to aggregate queries to be handled by for example databases with concurrent queuing (especially distributed ones where the processing can be fanned out to multiple servers)
i say, if you make the client and build out a relay interface go ahead, make your hipster CBOR encoder API
but for the love of God do not make up your own fancy new RPC API if you do not intend to support a sufficient range of common language bindings, NATIVELY... just take the json and binary encode it, don't fuck around
Yeah, protobuf is gRPC standard, like nostr:npub1wqfzz2p880wq0tumuae9lfwyhs8uz35xd0kr34zrvrwyh3kvrzuskcqsyn suggested.
i think in all this debate people just need to make a relay/client pair that understand the new protocol and done
I don't understand why they want to standardize this. Different devs will want different protocols and it's irrelevant to core interoperability because you can always communicate over json.
All I really care about is signed events that I can parse and verify with a npub. Plus a few conventions around kinds. If signed events can be serialized for more efficient transmission, great, but that’s icing on the cake.
Yup! Cryptographic identity and standardized data shapes are the most fundamental parts of Nostr. It's too hard to get devs to agree on everything else, so implementations will vary widely. If we keep to that core, though, we'll retain basic interoperability.
that's why i say make a binary encoder and runtime format for it like the ones i have designed
https://github.com/mleku/nostrbench
there is no way that anyone can make it faster than what i have done short of writing it in assembler, and it's such a short piece of code that it should be possible to implement it in any language
i am pretty sure that even javascript can deal with the binary 32 and 64 byte fields in the encoding so similar dramatic improvements in performance as you can see in those benchmarks should be possible, while also enabling a pure binary encoding and a binary detached hash and signature associated with it
just like fiatjaf made nostr json a custom thing instead of jsonrpc2 like bitcoin uses for RPC, we should have a custom binary codec just like what chain data uses on bitcoin
the hard part is going to be people who insist on javascript and python or the necessity of it for web apps, but even there, i am pretty sure you can make my codec into wasm modules and done
https://github.com/mleku/realy/blob/dev/event/binarymarshal.go and https://github.com/mleku/realy/blob/dev/event/binarymarshal.go are the in and out for the format, containing the ID and the signature of the json form
it's faster than fiatjafs and it's what i use in my database implementation
messagepack would work fine I think. biggest gains would be parsing efficiency and battery life gains. decoding json sucks and is slow.
nostrdb has optimizations to skip parsing altogether when it doesn’t need to (stops parsing at the id field if we already have the note). The performance boost there is nice. messagepack or some other format would be another boost.
The *ideal* way would be something like flatbuffers, where you could just memcpy the note into your db… but is more complex.
CBOR is basically the RFC standardized version of messagepack.
I recommend CBOR.
I proposed a format for that
my binary codec already does a lot of that memory remapping of fields, as the runtime and database versions of the data are basically the same - it keeps the id/pubkey/signature fields (including in common tags) in raw binary and unpacking it into the runtime format is just a matter of creating pointers
the hex encoding is also done with a SIMD hex encoder, and the sha256 hash is done also with a threaded worker based single instruction multiple data so, on avx512 and avx2 that means it runs 2 hashes per CPU thread
switching the binary fields to be kept as binary except up to the wire has a massive benefit
it is so nearly close to being a perfectly servicable wire codec as well, i just didn't design an envelope encoding or binary serialisation for the filters
but other languages probably won't support this kind of optimization very well certainly not javascript
i don't get how javascript parsing is really much slower working in native json (which should be optimized to the max) versus making javascript work with foreign binary data formats for these binary fields
but totally understand why it's hard to make clients that aren't JS/DOM/HTML/CSS based... the whole tech industry has focused on this universal platform and its abomination of a scripting language to the expense of serious languages supporting native apps, and the total lack of adequate multi-platform targeting and adequately broad language support for cocoa, win32, gtk and qt - ironically most of the time it's either electron, putting the front end in a browser engine, or some kind of simple and fairly inadequate immediate mode or similar direct opengl/vulkan based thing (eg imgui, egui, gio, fyne, nucular)
