For messaging, I’ve always been a bigger fan of exactly-what-you-need via zeromq or nanomsg. Kafka is fine, and AWS makes it easy, but I find it’s often weirdly opinionated and sometimes limiting. And fucking expensive unless you’re self hosting.
Discussion
We've got it running over 6 kubernetes nodes...
It's a monster.
Yep, that’s what I mean. As a thought experiment, consider how it could have been designed with direct peer to peer messaging via 0mq. You may have to do a bit of reading on it if you’re not familiar - it can seem like you’re having to build a lot from scratch. But it will only seem that way.
I’m curious if you might have found a lot of advantages this way.
The big issue you’ll run into is discoverability, but that’s easily managed by just updating a secrets list (or even just an s3 file) with the list of known listeners and producers.