Nostr

sure enough

and precisely where i suspected, in the tag encoder

gonna at least squash this bug anyway. this is a nasty one.

so, i think i have found it. it's a buffer being reused in the client library, this is why it didn't appear before, it's been some time since i was using the websocket client library.

the fix was simple: just stop reusing the buffer, make a new one for every received message. the end.

it was particularly affecting follow lists. it was also causing a race condition because the buffer was being used concurrently. with this one tiny change, allocating a new buffer for each message, the race condition has gone away and so has the mangled tags.

Reply to this note

Please Login to reply.

Discussion

ᴛʜᴇ ᴅᴇᴀᴛʜ ᴏꜰ ᴍʟᴇᴋᴜ 5mo ago

took a while to dig around and find the problem. it's partly related to the fact that the event and tag unmarshaler doesn't copy memory (which makes it faster) but this also makes concurrency cause funny problems.

what was literally happening was that an event was being decoded, and then the buffer holding the memory where an event was sliced out of, was then overwritten by the next incoming event from the relay the client was connecting to, and by the time the event was being saved to the database it had already been mangled. i think also the event ID, signature and pubkeys were being overwritten as well, because those also are zero copy operations.

lesson learned: zero copy codecs must be paired with fresh buffers from the network transport. otherwise you get data races.

now, i'm getting more in the frame of mind to think straight, and this is because i had a can of red bull. which in combination with the allergy messing with my brain suggests that eliminating the allergens will help me quit these bad habits. probably nicotine is also suppressing allergy symptoms on my brain. my brain is literally swelling and this is messing with my thinking, reducing my IQ dramatically.

glutens are the worst, second is caseins and especially caseins that have been ultrapasteurised, most likely unpasteurised natural milk would not do this. but it's what did it to me today. my brain has felt like mush.

so anyhow, next, i guess, to figure out how i need to write this publisher implementation for the single-listener design i have come up with. and then, with that, i think i can call the HTTP API an MVP. there is other methods i could optionally add, but i just want to put it to bed with a feature complete - compared to the websocket API, set of methods. import and export are orthogonal to this, but are essential features for administration. the event publish, filter search and subscription mechanisms make basic nip-01 feature parity.

i kinda promised myself that by the end of this month i would be starting on the job hunt for real. so, hopefully i can get this subscription system done in the next three days. preferably today, but with a mushy brain i'm probably not gonna get there that soon.