CODEWITHSUNDEEP
apache-zookeeperapache-kafka
dvl
dvl

Reputation: 761

Kafka offset out of sync

I have a 3 node kafka cluster with 1 zookeeper node.

Querying each broker for offsets is giving me different offsets for a partition (Partition 2). Broker 0 is stuck with a particular offset value for Partition 2. I have a few questions on this:

  1. Does this lag affect the producer?
  2. Can a consumer query a broker which is not leader for that partition?
  3. I have lost an ISR on this partition; how can I bring the cluster back to normal state? What metrics I can monitor to check if the node is catching up?

KAFKA version: 0.8.2.1

    /bin/kafka-run-class.sh kafka.tools.GetOffsetShell --topic AdServe  --broker-list broker0:9092 --time -1
    AdServe:0:25186452
    AdServe:1:26748318
    AdServe:2:15659873

    ./bin/kafka-run-class.sh kafka.tools.GetOffsetShell --topic AdServe  --broker-list broker1:9092 --time -1
    AdServe:0:25186652
    AdServe:1:26748318
    AdServe:2:19100186

    ./bin/kafka-run-class.sh kafka.tools.GetOffsetShell --topic AdServe  --broker-list broker2:9092 --time -1
    AdServe:0:25187252
    AdServe:1:26748318
    AdServe:2:19103185


Topic status on zookeeper:

    Topic:AdServe   PartitionCount:3    ReplicationFactor:2 Configs:
        Topic: AdServe  Partition: 0    Leader: 1   Replicas: 1,0   Isr: 0,1
        Topic: AdServe  Partition: 1    Leader: 2   Replicas: 2,1   Isr: 2,1
        Topic: AdServe  Partition: 2    Leader: 2   Replicas: 0,2   Isr: 2

Upvotes: 2

Views: 8090

Answers (1)

Jon Hunter
Jon Hunter

Reputation: 880

I found that restarting the lead broker for that topic sorted out the ISR.

This article was really useful on understanding Kafka replication and how to monitor if topics are in sync

https://www.confluent.io/blog/hands-free-kafka-replication-a-lesson-in-operational-simplicity/

Upvotes: 7

Related Questions