CODEWITHSUNDEEP
apache-kafkakafka-producer-api
eatSleepCode
eatSleepCode

Reputation: 4637

Kafka: Exception while producing records on Kafka topic

I am getting below exception while producing records on Kafka topic: 

java.lang.RuntimeException: This server is not the leader for that topic-partition.

Below is the code for sending records to Kafka topic.

AtomicReference<Exception> exRef = new AtomicReference<>();
while([some condition]) {
    producer.send(new ProducerRecord<>(topic, message), (metadata, exception) -> {
        if (exception != null) {
            exRef.set(exception);
        }
    });
    if (exRef.get() != null) {
        throw new RuntimeException(exRef.get().getMessage(), exRef.get().getCause());
    }
}

I just need to know what is the reason for this exception? How to prevent it?

Upvotes: 0

Views: 1605

Answers (1)

Matthias J. Sax
Matthias J. Sax

Reputation: 62350

Each topic partitions has a leader broker that is responsible to serve all read/write request from clients (and if replication is larger than one, multiple follower brokers that only copy the data from the leader broker for this partitions but don't serve any read/write request from clients). On startup of a client, the client queries the cluster to get the leader for each partitions it needs to read/write from/to and caches this metadata.

It can happen, that the leadership of a topic partitions is moved from one broker to another (ie, a follower becomes a leader) for different reasons (for example, the broker goes down -- after recovery it might not be the leader anymore, but be a follower as there is no need to transfer leadership back -- or if a admin command is issued, to move the partition to a different broker).

If this happens, the client's metadata is not correct any more and you get the corresponding exception. Thus, from a client point of view, you cannot prevent this from happening. However, you can simple create a new client instance that will rediscover the new leader and your app can resume from there.

Upvotes: 3

Related Questions