CODEWITHSUNDEEP
jmsrabbitmqmessagingapache-kafka
WickedElephant
WickedElephant

Reputation: 347

Is Kafka able to have a dynamic number of consumers?

We are looking for a new messaging platform, and have narrowed our choices down to RabbitMQ or Kafka.

Right now, I am leaning toward Kafka, but I have some doubts that it is a good choice given one of our requirements.

We need to have a queue that is consumed by an unknown number of consumers. That is, we need to dynamically add and remove consumers as "workers" come online to do the processing. Also, workers may drop off at any time.

So for example, we may start a queue that has no consumers at all, and then the number of consumers may grow to 30. Later it may grow to 5000 or more, and then drop back off to 3.

We do not care about message ordering for this particular use case. Is Kafka a good fit for this?

Also, we were planning on maintaining a pool of consumer threads so that the workers could grab a single message and process it. So there may be 100 consumers in the pool and only 20 workers. Is it possible that we end up with messages in the other 80 consumers which are not utilized in the workers due to message send buffering? In other words, does Kafka pre-deliver messages to consumers before they are requested like some messaging systems do?

Upvotes: 2

Views: 403

Answers (1)

codejitsu
codejitsu

Reputation: 3182

Yes, kafka can definitely match your requirements. You can have many-to-many producers/consumers. If all your consumers are within the same consumer group all messages will be distributed evenly between all consumers. It is not a problem also if you shut down / add new consumers, kafka will manage all automatically for you.

To your last question - kafka consumers are pull-based, so it is consumer responsibility to check if there are some messages to process.

Upvotes: 1

Related Questions