CODEWITHSUNDEEP
message-queueproducer-consumerapache-kafkaworkerkafka-consumer-api
java_geek
java_geek

Reputation: 18035

Kafka Consumers Work Queue

We have the following scenario in an SOA application. ServiceA produces some jobs which need to be processed by instances of ServiceB asynchronously. Essentially, this translates to a Work Queue problem where each worker is an instance of ServiceB. We are using Kafka as the message broker and have the following setup.

5 Brokers B1,B2,B3,B4 and B5. There is a topic(A) with 10 partitions (P1,P2,....P10) each with a replication factor of 3. Lets assume that the partition assignment is as follows P(i) has B(i) as leader and B(i+1) and B(i+2) as replicas.

There are 3 instances of ServiceB running. Given this setup, how should we use the High Level Consumer API to achieve the consumption model C1 consumes from 3 partitions; C2 from 3 partitions; C3 from remaining 4 partitions

Upvotes: 0

Views: 402

Answers (1)

Gwen Shapira
Gwen Shapira

Reputation: 5168

You don't control things to this level using the high level consumer.

If consumers C1, C2 and C3 are part of the same consumer group (i.e. have same group.id), the high level consumer will attempt to balance partitions between them. You should end up with 2 consumers each consuming from 2 partitions and one reading from 4 partitions. However you don't control the specifics.

If you need more control over which consumer gets which number of partitions, you'll need to do it yourself with the simple consumer API.

Upvotes: 1

Related Questions