CODEWITHSUNDEEP
apache-kafkakubernetesapache-zookeeper
Assem
Assem

Reputation: 12107

Zookeeper refuses Kafka connection from an old client

I have a cluster configuration using Kubernetes on GCE, I have a pod for zookeeper and other for Kafka; it was working normally until Zookeeper get crashed and restarted, and it start refusing connections from the kafka pod:

Refusing session request for client /10.4.4.58:52260 as it has seen zxid 0x1962630

The complete refusal log is here:

2017-08-21 20:05:32,013 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /10.4.4.58:52260
2017-08-21 20:05:32,013 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@882] - Connection request from old client /10.4.4.58:52260; will be dropped if server is in r-o mode
2017-08-21 20:05:32,013 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@901] - Refusing session request for client /10.4.4.58:52260 as it has seen zxid 0x1962630 our last zxid is 0xab client must try another server
2017-08-21 20:05:32,013 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008] - Closed socket connection for client /10.4.4.58:52260 (no session established for client)

Upvotes: 9

Views: 25686

Answers (4)

Dave
Dave

Reputation: 73

I had this problem. This is how I solved it. first I shut down both zookeeper and kafka then I started zookeeper, then I started kafka.

This worked on Centos 7. It assumes you are one level above the bin directory that the kafka scripts are in. I used 2 windows, one for the zookeeper process and one for the kafka process. This was an experiment to see what would work and what wouldnt. You maybe want to have things run in the background and redirect the output to a log file for use in production etc....

the exact steps:

 1. shut down kafka and zookeeper
    ./bin/kafka-server-stop.sh  ./bin/zookeeper-server-stop.sh 
    
 2. start zookeeper
    ./bin/zookeeper-server-start.sh ./config/zookeeper.properties
    
 3. start kafka 
    ./bin/kafka-server-start.sh ./config/server.properties

Upvotes: 1

Asanke
Asanke

Reputation: 601

Related to the answer from @GuangshengZuo.... Steps

  1. Stop any residual zookeeper instances - zookeeper-server-stop.bat
  2. Start a fresh zookeeper- zookeeper-server-start.bat .\config\zookeeper.properties This will do

Upvotes: 2

lmo
lmo

Reputation: 547

For the record, I had this problem and all my kafka were off.
But, my kafka-manager was still up and listening on zookeepers. Turning it off resolved the issue.

Upvotes: 4

GuangshengZuo
GuangshengZuo

Reputation: 4687

Because the kafka maintain a zookeeper session which remember the last zxid it has seen. So when the zookeeper sevice go down and come again, the zk's zxid begin from a smaller value. and ZKserver think the kafka has seen a bigger zxid, so it refuse it.

Have a try to restart the kafka.

Upvotes: 20

Related Questions