CODEWITHSUNDEEP
apache-kafkakafka-python
M. Carlson
M. Carlson

Reputation: 786

Mulitple Python Consumer Threads on a Single Partition with Kafka 0.9.0

For context, I am trying to transfer our python worker processes over to a kafka (0.9.0) based architecture, but I am confused about the limitations of partitions with respect to the consumer threads. Will having multiple consumers on a partition cause the other threads on the same partition to wait for the current thread to finish?

Upvotes: 0

Views: 868

Answers (1)

Marko Bonaci
Marko Bonaci

Reputation: 5708

I'm actually not sure for Kafka 0.9, haven't yet had the need to go over the new design thoroughly, but AFAIK this wasn't possible in v8.
It certainly wasn't possible with the low-level consumer, but I also think that, if you assign more threads than you have partitions in the high-level consumer, only one thread per partition would be active at any time. This is why we say that parallelism in Kafka is determined by the number of partitions (which can be dynamically increased for a topic).

If you think about it, that would require coordination on the message level between the consuming threads, which would be detrimental to performance. Consumer groups in v0.8 were used to make the thread -> partition assignment a responsibility of Kafka, not to coordinate multiple threads over a single partition.

Now, it could be that this changed in 0.9, but I doubt that very much.

[EDIT] Now that I'm reading your question once again, I hope I understood your question correctly. I mean, having multiple consumers (not consumer threads) per partition is a regular thing (each has its own offset), so I assumed you were asking about threads/partitions relationship.

Upvotes: 1

Related Questions