CODEWITHSUNDEEP
apache-kafkakafka-consumer-api
Nicolas El Khoury
Nicolas El Khoury

Reputation: 6507

how to automatically delete unused kafka consumers

I am using Kafka for a messaging application. For this application, there is a producer putting messages into a topic, and consumers registered to this topic, and consuming these messages. These consumers are Dockerized applications. For autoscaling purposes, each consumer, upon its creation, is registered as a consumer with a unique ID.

Assume the following scenario:

Consumer1 is created as a docker container, and registers itself as a consumer with ID Consumer1

Consumer2 is created as a docker container, and registers itself as a consumer with ID Consumer2

Now for whatever reason Consumer1 fails, and gets replaced by Consumer3 which registers itself as a consumer to kafka with an ID of Consumer3.

The problem is, Consumer1 is no longer used. On the long term, there will be multiple unused consumers.

Is there a way to dynamically and automatically know which consumers are no longer used and delete them?

Upvotes: 1

Views: 456

Answers (1)

Steephen
Steephen

Reputation: 15824

If consumer1 and consumer3 belongs to the same consumer group, consumer3 will start reading messages from where consumer1 left off. This is because Kafka maintains the offset specific to a consumer group. So in case one among the consumers with same consumer group fails, others will use the offset and avoid reprocessing the data.

Kafka broker does not maintain the failed consumers log anywhere as you assume in your question.

Upvotes: 0

Related Questions