Partitioning is probably the best idea.
I’m thinking we need a binary version of Nostr events (not binary encoded events), for cases where it’s not complicated enough to involve a system like Blossom, but also would benefit from binary encoding like encrypted events.
Explain what you mean by binary version vs binary events. Not sure I’m following
Binary encoding: we keep compatibility with nostr events, just optimize JSON field names away
Binary events: breaking change in signing scheme that also means conversion to JSON is optional, tag values and event contents can now be binary
s/optional/not needed/
We can have alternative signatures in a specific parameterized single letter tag
Labeling events could be used to augment missing tags after the fact
IMO a common precompression dictionary would be even better and simpler
The canonical form, also, is better, there really is no need to transmit the event id
why?
we can just add a new field like “binarySig”, relays can validate both of them (and backfill binary signatures if later added by the author), and clients can pick one
yeah, that's what i'm saying but just make it a tag so clients can generate it directly as well
i would prefer it if there was no new fields in events
as for filters i have said that "it would be useful to have a negation operator, eg !"
the problem is with a tag you can’t add the faster signature later
why not? tags are just lists of strings?... i can picture `["b","
I was thinking you would add it to the original event
you can't do that... it's a chicken and egg problem
if you are going to make a canonical binary encoding, exactly the same as the json canonical you do not hash on the hash (obviously) or the signature (also obviously) but when i say obviously, you don't realise that until you think about it
the ID and signature are external to the event, even if they appear in the naive version sent over the wire, that is only there really for fast access, a convenience, but to make the hash again you remove the signature and ID from the event and remove the object keys and replace it with a strict faithful ordering as received otherwise
the same rules can be applied to a binary encoding, so a hypothetical "b" tag could be added with other variations but if the signature is missing that is not an available verification method, so fall back to the json canonical for that, as the signature field is also present there, and would be retained in a binary wire format for the same reason (legacy, perhaps, but probably it will never go away, and good i say)
so, yeah, in summary IDs and Signature fields are inherently external to the data, thus they can be added or removed with impunity and if we were to propose a NIP to use them, this makes it simpler to explain, "b tags must not be added in the canonical form to derive the json ID hash"
in this way the event can be sent on the wire with this binary signature on it, enabling the forming of the binary encoding corresponding to it later, but it has to be omitted
so, yeah, maybe that's too complicated for people's brains, but ultimately that's how it would have to work
also, just should point out, that the wire format could actually be the canonical encoding, it's not that wasteful of space really, already eliminates all the repetitive key strings and all
and a binary version of the canonical encoding also would leave out the event IDs and only retain the signatures
anyway, the bigger struggle with this is the morons who think shit like CBOR and MSGPACK and Protobuf etc are more important debate materials
they really aren't, and i think that there really is a lot of justification for the idea of just making one binary encoding canonical form that gives you a second signature field and done
any programming language doing binary RPC should be able to deal with it, i don't think it's even that hard in javascript to decode binary data and unpack it into JSON and import it
Apparently I'm one of those morons. Why wouldn't you want to use CBOR?
Is your solution just to binary encode most of the json event to squish it down but then keep an id/sig (and maybe a tag) unencoded? I'm not sure I'm following what you're suggesting. There's too much vitriol mixed in with the ideas. 😅
I’ll DM you an explanation for my idea that will end up never finished soon. 😂
don't need the ID there at all in the wire format, and it's only needed as an index in the database
you would have vitriol too if you looked too close at the code to see how much the politics influences the architecture in a very negative way
also, you would also have vitriol if people respond by focusing on the person instead of the engineering, so, please, keep it on topic
and answering the technical part of it
- compressing hex is, you know, rather simple, 50% if you know the field is hex
- a fixed ordering of fields instead of field names like in the JSON eliminates the need for labeling, it's literally just a set of offsets and length prefixes
- where you know the field is going to be binary, you can encode it as binary, in the expected length, i have used this in my runtime and database formats in my code - and there is a lot of places with this, field 2 in e and p tags, the decimal:pubkeyhex:string of a tags, the ID (which can be omitted) and the pubkeys and signatures all can be sent over the wire as binary, without any length prefix if they obey the ordering
bringing your opinions about programming languages and this RFC and that RFC is stupid we have options here we don't need to hitch our horse to some anonymous committeit over in Buffalo or Antwerp we are building our own protocol and we can easily come to agreement on the performance characteristics and limitations of implementations thanks to our diversity
fiatjaf was right in the way he designed the scheme pretty much, except on a few small points like why send the event in object format when you could have just sent the canonical form that means you literally can hash it on the spot and get the ID? verify the signature and two steps just got eliminated
i kindly invite you to go inspect the protocol specification of Bluesky and come back to me after you have digested the abomination for your opinions about teh use of CBOR
yes, I’m also saying the new binary signature is another detached field like id/sig
but how to preserve it in the json object so the binary version can be derived/verified from it? easier to stick with existing fields but define a strict meaning to one that is then essentially exempted from the indexing rule, this way clients won't break decoding it even if they don't use it
adding more keys to the object is going to break them, and you do want to keep these extra fields (sans ID) in other encodings
yes, my suggestion would be then, that you put the tags so it's `["b","
we don't have to have the ID in the wire format, period, so it doesn't have to be in the tag either, actually
i personally am very much in favour of making a gradual move towards this by suggesting that relays and clients with some nip-11 flag understand events that are in canonical format, and then you can add a futher tag with a list of supported encoders and the in-event signature for other formats, which are optional, only the json form is mandatory, can have signatures for your faster verification form the binary form. done.
also just need to point out - the signer of an event could make an after-the-fact amendment to add the tag with another tag that refers to the replaced version that gives you your new (multiple) signature tags as i described, much like the parameterised replaceable protocol already existing
Ok - this might be left curve but... Why would we not just encode using CBOR, then attach a signature. 🤷♂️ Seems like that's all we'd need.
Relays could destructure however they want for indexing but the data structure and event creation/encode/decode would be super simple.
all this has me thinking, this is something that nip-32 labels could be used for - providing the signatures for alternative encoding formats, retroactively, without altering the original event
Interesting. I think if we’re going to ever take the hit on changing anything to binary we should change it all to a binary format.
