CODEWITHSUNDEEP
apache-kafkakafka-producer-api
Chantz
Chantz

Reputation: 5963

What is the difference between partitions & available partitions for topic in Kafka?

I am planning to write my own Partitioner for Kafka Producer, so I was seeing the implementation of Kafka's DefaultPartitioner.

I saw that it calls Cluster's availablePartitionsForTopic and sometimes calls partitionsForTopic for computing partition.

I read the documentation & also saw the source code, but I am not able to see what is the difference between the two.

Can someone point me to the right documentation for this or explain the difference?

Upvotes: 2

Views: 658

Answers (2)

senseiwu
senseiwu

Reputation: 5259

To answer the question difference between partitionsForTopicand availablePartitionsForTopic (not how DefaultPartitioner uses them to assign a Partition), code is the only documentation

Have a look at org.apache.kafka.common.Cluster, 

    this.partitionsByTopic = new HashMap<>(partsForTopic.size());
    this.availablePartitionsByTopic = new HashMap<>(partsForTopic.size());
    for (Map.Entry<String, List<PartitionInfo>> entry : partsForTopic.entrySet()) {
        String topic = entry.getKey();
        List<PartitionInfo> partitionList = entry.getValue();
        this.partitionsByTopic.put(topic, Collections.unmodifiableList(partitionList));
        List<PartitionInfo> availablePartitions = new ArrayList<>();
        for (PartitionInfo part : partitionList) {
            if (part.leader() != null)
                availablePartitions.add(part);
            }
        this.availablePartitionsByTopic.put(topic, Collections.unmodifiableList(availablePartitions));
    }

As you can see, the distinguishing factor between the two is the availability of a leader

Upvotes: 2

amethystic
amethystic

Reputation: 7079

If you specify a key for a record, Kafka probably thinks you definitely want this record to be sent to some determined partition even if it is not available at that time.

However, if no key is specified, then Kafka might think you care nothing about the target partition the record goes to, so it picks up one randomly from those "alive" partitions.

Upvotes: 4

Related Questions