CODEWITHSUNDEEP
apache-kafkakafka-consumer-api
Janani
Janani

Reputation: 35

All Consumers in Kafka Consumer group to consume all Produced records

When we create a consumer group to consume a set of records from producer, the consumers within consumer group will consume the produced records in parallel, I see this as default behaviour.

Is there a way to consume these sequentially? For example If Producer produced 1,2,3,4,5,6,7,8,9,10 records.

By default within a consumer group, consumer1 might consume 1,3,5,7,9 and consumer2 might consume 2,4,6,8,10.

Can this be changed in a way that, in a consumer group we have consumer1 consuming 1,2,3,4,5,6,7,8,9,10 records and consumer2 consuming 1,2,3,4,5,6,7,8,9,10 records?

I am new to Kafka and trying to learn. In this process of learning, tried out the default behaviour which is working as expected. Looking forward to understand if we can consume all records produced as part of single consumer group.

Upvotes: 0

Views: 183

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191738

consumer1 might consume 1,3,5,7,9 and consumer2 might consume 2,4,6,8,10

Only if there's two partitions in the topic, and records are evenly distributed... This is not the default behavior (ref docs on StickyPartitioner)

Can this be changed in a way that, in a consumer group we have consumer1 consuming 1,2,3,4,5,6,7,8,9,10 records and consumer2 consuming 1,2,3,4,5,6,7,8,9,10 records?

Not in the same consumer group, no. At least, not from the same topic. You could use MirrorMaker2, for example, and effectively duplicate your topic, thereby storing and consuming the data twice.

Alternatively, use assign API rather than subscribe, then groups aren't used and all consumers can read all assigned partitions without coordination

Upvotes: 0

Related Questions