CODEWITHSUNDEEP
partitioningapache-kafkaproducer-consumer
Abhishek bhutra
Abhishek bhutra

Reputation: 1422

Kafka have both partitioning and consumer group

Can we use both partitioning and consumer group for same topic. So we want to create a topic which have partitioning and then create multiple consumers to it and out of them 2-3 will be generic listening to all messages(needs to be in a consumer group so that message in not processed multiple times) and then one consumer for specific partion.

Is partitioning and consumer are mutually exclusive?

Upvotes: 0

Views: 875

Answers (2)

Anatoly Deyneka
Anatoly Deyneka

Reputation: 1258

High Level Consumer for kafka 0.8.x doesn't allow you to specify partition. It reads the data from all partitions and does complex failure detection and rebalancing. Probably it will be supported in future versions according to the API redesign - https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design#ConsumerClientRe-Design-Allowmanualpartitionassignment.

If you need to read from specific topic and partition - use SimpleConsumer(it requires manual leader/offset/exclusivity/error handling).

Or you can use high level consumer and filter the data by partition(if you accept the overhead).

Also as an option you can redesign your topics to write data to separate topic instead of 'specific partition'.

Upvotes: 1

Lundahl
Lundahl

Reputation: 6572

With the high level consumer API you can´t pin a consumer instance to a particular partition but there is nothing preventing you from having a set of consumers using the high level API and another set of consumers using the simple API for the same topic.

With this you could have a simple consumer consuming from a specific partition and a set of high level consumers in a consumer group consuming messages across all partitions.

Upvotes: 1

Related Questions