CODEWITHSUNDEEP
pythonapache-kafkakafka-consumer-api
Soufiane Roui
Soufiane Roui

Reputation: 679

Having same number of consumers as the number of partitions

I have a Kafka topic with 4 partitions, and I am creating an application written with python that consumes data from the topic.

My ultimate goal is to have 4 Kafka consumers within the application. So, I have used the class KafkaClient to get the number of partitions right after the application starts, then, I have created 4 threads, each one has the responsibility to create the consumer and to process messages.

As I am new to Kafka (and python as well), I don't know if my approach is right, or it needs enhancements (e.g. what if a consumer fails).

Upvotes: 0

Views: 282

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191743

If a consumer thread dies, then you'll need logic to handle that.

Threads would work (aiokafka or Faust might be a better library for this), or you could use supervisor or Docker orchestration to run multiple consumer processes

Upvotes: 1

Related Questions