Nostr

i've got a task set for myself currently, to implement a HTTP SSE based subscription system, and it's different in how it works to my previous implementation. the previous implementation created one SSE connection for every filter, and i need to make one now that has one SSE connection and an identifier of this channel is sent as the first message and the client then uses this with a subscribe method that uses this identifier to specify which channel the events should be directed to.

it's a lot different and i really need to totally rewrite the publisher implementation to fit the task, just copypasta of the old version is not going to work because i can't make sense of how this changes how the publisher should work.

the main reason i have to do it is because of old limitations on HTTP/1 that only allow max 6 such connections to a domain within the entire web browser, which is not going to work reliably and is a difficult gotcha for client developers.

there seems to be some bug somewhere in the tag decoding somewhere, it looks like there is a concurrent overwriting of buffers that are picked apart to become the event's tag arrays. i probably should just shelve this current set of changes, which i have failed to do anything with since i started working on them ... i think the day before yesterday... and do some bug hunting. i think i need to specifically catch race conditions, to find where the bug is happening.

ᴛʜᴇ ᴅᴇᴀᴛʜ ᴏꜰ ᴍʟᴇᴋᴜ 5mo ago

sure enough

and precisely where i suspected, in the tag encoder

gonna at least squash this bug anyway. this is a nasty one.

Reply to this note

Please Login to reply.

Discussion

ᴛʜᴇ ᴅᴇᴀᴛʜ ᴏꜰ ᴍʟᴇᴋᴜ 5mo ago

so, i think i have found it. it's a buffer being reused in the client library, this is why it didn't appear before, it's been some time since i was using the websocket client library.

the fix was simple: just stop reusing the buffer, make a new one for every received message. the end.

it was particularly affecting follow lists. it was also causing a race condition because the buffer was being used concurrently. with this one tiny change, allocating a new buffer for each message, the race condition has gone away and so has the mangled tags.

ᴛʜᴇ ᴅᴇᴀᴛʜ ᴏꜰ ᴍʟᴇᴋᴜ 5mo ago

took a while to dig around and find the problem. it's partly related to the fact that the event and tag unmarshaler doesn't copy memory (which makes it faster) but this also makes concurrency cause funny problems.

what was literally happening was that an event was being decoded, and then the buffer holding the memory where an event was sliced out of, was then overwritten by the next incoming event from the relay the client was connecting to, and by the time the event was being saved to the database it had already been mangled. i think also the event ID, signature and pubkeys were being overwritten as well, because those also are zero copy operations.

lesson learned: zero copy codecs must be paired with fresh buffers from the network transport. otherwise you get data races.

now, i'm getting more in the frame of mind to think straight, and this is because i had a can of red bull. which in combination with the allergy messing with my brain suggests that eliminating the allergens will help me quit these bad habits. probably nicotine is also suppressing allergy symptoms on my brain. my brain is literally swelling and this is messing with my thinking, reducing my IQ dramatically.

glutens are the worst, second is caseins and especially caseins that have been ultrapasteurised, most likely unpasteurised natural milk would not do this. but it's what did it to me today. my brain has felt like mush.

so anyhow, next, i guess, to figure out how i need to write this publisher implementation for the single-listener design i have come up with. and then, with that, i think i can call the HTTP API an MVP. there is other methods i could optionally add, but i just want to put it to bed with a feature complete - compared to the websocket API, set of methods. import and export are orthogonal to this, but are essential features for administration. the event publish, filter search and subscription mechanisms make basic nip-01 feature parity.

i kinda promised myself that by the end of this month i would be starting on the job hunt for real. so, hopefully i can get this subscription system done in the next three days. preferably today, but with a mushy brain i'm probably not gonna get there that soon.