Replying to Avatar Blake

I’d use a queue for inbound events and then you can pull from that queue using a worker type flow and then add the event id to a set, after doing a membership test to see if you’ve already processed the event.

Alternatively, since the set will grow in size forever (between restarts), you’ll either need to persist the set data (like Redis) or use a LRU cache or something like a ring buffer. Or perhaps a bloom filter if an occasional false positive is ok.

It all kind of depends during what time frame do you expect the duplicate events. Over an hour window, or days, or months, etc. Is reprocessing an event forbidden. And how many events or relay sources.

Avatar
Rich Nost 2y ago

Literal seconds, I would imagine.

Reply to this note

Please Login to reply.

Discussion

No replies yet.