CODEWITHSUNDEEP
apache-kafkakafka-consumer-apikafka-producer-api
senseiwu
senseiwu

Reputation: 5259

Kafka topic partition zero is not getting used

I created a topic with 3 partitions

Topic:fixtures  PartitionCount:3    ReplicationFactor:1 Configs:
    Topic: fixtures Partition: 0    Leader: 0   Replicas: 0 Isr: 0
    Topic: fixtures Partition: 1    Leader: 0   Replicas: 0 Isr: 0
    Topic: fixtures Partition: 2    Leader: 0   Replicas: 0 Isr: 0

I use a Java Producer to write into the topic synchronously 

producer.send(new ProducerRecord<>(KafkaProperties.TOPIC_FIXTURES, key, value)).get();

I have a consumer in Java which subscribes and reads from it

Consumer<String, String> kafkaConsumer = new KafkaConsumer<>(configs);
kafkaConsumer.subscribe(Collections.singletonList(KafkaProperties.TOPIC_FIXTURES));

My keys are always a fixed set of 3 different strings (k1,k2,k3). But my messages are always going to either partition 1 or partition 2 -- k1 and k2 goes to partition 1 and k3 goes to partition 2.

Why is partition 0 unused?

Upvotes: 2

Views: 5837

Answers (1)

Mickael Maison
Mickael Maison

Reputation: 26885

For keyed messages, the default partitioner computes the partition based on a hash of the key.

With many keys, using a hash of the key is an efficient method to spread messages evenly across partitions.

But if you only have very few different keys (or if the number of keys is close to the number of partitions), it's possible it won't use all the partitions.

If that's the case, then you can provide your own partitioner to the Producer using the partitioner.class configuration to do round-robin for example.

Upvotes: 4

Related Questions