Spent my day working on some data engineering pipelines that make use of rabbitMQ. It really is such a versatile queuing system when you need more than the simple queues/streams redis can offer and aren’t pushing super high volumes that require something like Kafka. One of the most configurable message brokers out there 👌
Discussion
I use MQTT at home and AWS SQS at work. Both of which are pretty basic. What are some features of Rabbit MQ that you take advantage of?
One of the things I really enjoy about rabbit is the fan out pattern that it supports. One job can publish events that N number of other jobs can subscribe to. These consumers can be entirely decoupled, with multiple workers - each consumer can get the same events (redis streams support something similar, but they don’t support ack/nack and handling dropped messages isn’t as graceful as it is with rabbit) and a consumer can subscribe to more than one queue at the same time. It also supports routing keys that can make distributing messages even more configurable, as well as dead letter queues that can be used to route errored or timed out events to other queues that can have their own set of consumers. Instead of having to spin up resources in terraform for new queues/consumers, each job can self register with the rabbit cluster programmatically and configure itself with its own specific filters. It can be consumed from both in blocking and non blocking ways, and supports auto nacking with heartbeats to ensure messages don’t get missed.
Redis is working fine for me atm, but I see a rabbitMQ migration if Nostr continues to grow - especially for multi-region architecture.
Yeah, I love redis, too. Definitely my go to for centralized in memory caching, and I usually start with it for basic queues as well (since most projects I have already use it). There’s just a complexity point where rabbit starts to make a bit more sense. But they actually work really well together.