Message queue (like RabbitMQ) or Kafka for Microservices?

Question

We are starting a new project, where we are evaluating the tech stack for asynchronous communication between microservices? We are considering RabbitMQ and Kafka for this.

Can anyone shed some light on the key considerations to decide one between these twos?

Thanks

Answer 1

Kafka	Rabbit MQ
It's a distributed streaming platform, working on the pub-sub model.	It's a message-broker, that works on pub-sub and queue-based model.
No out of the box support for retries and DLQ	Supports retries and DLQ out of the box(via DLX).
Consumers can't filter messages specifically.	Topic exchange and header exchange facilitate consumer-based message filtering.
Messages are retained till their validity period.	Messages are gone as soon as they are read.
Does not support scheduled or delayed message routing.	Supports scheduled and delayed routing of messages.
Scales horizontally	Scales vertically
Pull based approach	Pull based apporach
Supports event replay with consumer groups	No way to event replay

Answer 2

In order to select a message broker, I think this list could be really helpful.

 Supported programming languages: You probably should pick one that supports a variety of programming languages.

 Supported messaging standards: Does the message broker support any standards, such as AMQP and STOMP, or is it proprietary?

 Messaging order: Does the message broker preserve the ordering of messages?

 Delivery guarantees: What kind of delivery guarantees does the broker make?

 Persistence: Are messages persisted to disk and able to survive broker crashes?

 Durability: If a consumer reconnects to the message broker, will it receive the messages that were sent while it was disconnected?

 Scalability: How scalable is the message broker?

 Latency: What is the end-to-end latency?

 Competing consumers: Does the message broker support competing consumers?

Answer 3

Selection depends upon what exactly your microservices needs. Both has something different as compared to other.

RabbitMQ in a nutshell

Who are the players:

1. Consumer
2. Publisher
3. Exchange
4. Route

The flow starts from the Publisher, which send a message to exchange, Exchange is a middleware layer that knows to route the message to the queue, consumers can define which queue they are consuming from (by defining binding), RabbitMQ pushes the message to the consumer, and once consumed and acknowledgment has arrived, message is removed from the queue. Any piece in this system can be scaled out: producer, consumer, and also the RabbitMQ itself can be clustered, and highly available.

Kafka

Who are the players

1. Consumer / Consumer groups
2. Producer
3. Kafka source connect
4. Kafka sink connect
5. Topic and topic partition
6. Kafka stream
7. Broker
8. Zookeeper

Kafka is a robust system and has several members in the game. but once you understand well the flow, this becomes easy to manage and to work with. Producer send a message record to a topic, a topic is a category or feed name to which records are published, it can be partitioned, to get better performance, consumers subscribed to a topic and start to pull messages from it, when a topic is partitioned, then each partition get its own consumer instance, we called all instances of same consumer a consumer group. In Kafka messages are always remaining in the topic, also if they were consumed (limit time is defined by retention policy) Also, Kafka uses sequential disk I/O, this approach boosts the performance of Kafka and makes it a leader option in queues implementation, and a safe choice for big data use cases.

Use Kafka if you need

1. Time travel/durable/commit log
2. Many consumers for the same message
3. High throughput
4. Stream processing
5. Replicability
6. High availability
7. Message order

Use RabbitMq if you need:

1. flexible routing
2. Priority Queue
3. A standard protocol message queue

For more info