Kafka Failing to Start

Question

On my local Ubuntu 20.04, Kafka is failing to start.

kafka@bablo-HP-ProBook-440-G5:~$ sudo systemctl status kafka
● kafka.service
     Loaded: loaded (/etc/systemd/system/kafka.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Wed 2021-03-03 12:10:36 IST; 3min 26s ago
    Process: 32197 ExecStart=/bin/sh -c /home/kafka/kafka/bin/kafka-server-start.sh /home/kafka/kafka/config/server.properties > /home/kafka/>
   Main PID: 32197 (code=exited, status=1/FAILURE)

Mar 03 12:10:33 bablo-HP-ProBook-440-G5 systemd[1]: Started kafka.service.
Mar 03 12:10:36 bablo-HP-ProBook-440-G5 systemd[1]: kafka.service: Main process exited, code=exited, status=1/FAILURE
Mar 03 12:10:36 bablo-HP-ProBook-440-G5 systemd[1]: kafka.service: Failed with result 'exit-code'.

On the same host the zookeeper service is running smoothly.

kafka@bablo-HP-ProBook-440-G5:~$ sudo systemctl status zookeeper
● zookeeper.service
     Loaded: loaded (/etc/systemd/system/zookeeper.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2021-03-03 11:47:27 IST; 28min ago
   Main PID: 932 (java)
      Tasks: 38 (limit: 9359)
     Memory: 84.1M
     CGroup: /system.slice/zookeeper.service
             └─932 java -Xmx512M -Xms512M -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCI>

Mar 03 11:48:37 bablo-HP-ProBook-440-G5 zookeeper-server-start.sh[932]: [2021-03-03 11:48:37,793] INFO Created server with tickTime 3000 minS>
Mar 03 11:48:38 bablo-HP-ProBook-440-G5 zookeeper-server-start.sh[932]: [2021-03-03 11:48:38,022] INFO Using org.apache.zookeeper.server.NIOS>
Mar 03 11:48:38 bablo-HP-ProBook-440-G5 zookeeper-server-start.sh[932]: [2021-03-03 11:48:38,037] INFO Configuring NIO connection handler wit>
Mar 03 11:48:38 bablo-HP-ProBook-440-G5 zookeeper-server-start.sh[932]: [2021-03-03 11:48:38,058] INFO binding to port 0.0.0.0/0.0.0.0:2181 (>
Mar 03 11:48:38 bablo-HP-ProBook-440-G5 zookeeper-server-start.sh[932]: [2021-03-03 11:48:38,265] INFO zookeeper.snapshotSizeFactor = 0.33 (o>
Mar 03 11:48:38 bablo-HP-ProBook-440-G5 zookeeper-server-start.sh[932]: [2021-03-03 11:48:38,279] INFO Snapshotting: 0x0 to /tmp/zookeeper/ve>
Mar 03 11:48:38 bablo-HP-ProBook-440-G5 zookeeper-server-start.sh[932]: [2021-03-03 11:48:38,291] INFO Snapshotting: 0x0 to /tmp/zookeeper/ve>
Mar 03 11:48:38 bablo-HP-ProBook-440-G5 zookeeper-server-start.sh[932]: [2021-03-03 11:48:38,406] INFO Using checkIntervalMs=60000 maxPerMinu>
Mar 03 11:48:45 bablo-HP-ProBook-440-G5 zookeeper-server-start.sh[932]: [2021-03-03 11:48:45,679] INFO Creating new log file: log.1 (org.apac>
Mar 03 11:57:26 bablo-HP-ProBook-440-G5 zookeeper-server-start.sh[932]: [2021-03-03 11:57:26,915] WARN fsync-ing the write ahead log in SyncT>
lines 1-19/19 (END)

I checked the whether the port 9092 isn't preoccupied, it's not.

kafka@bablo-HP-ProBook-440-G5:~$ sudo lsof -i:9092
kafka@bablo-HP-ProBook-440-G5:~$ 

I am not getting anything as output of this command.

When I opened the server.log in the /home/kafka/logs. I noticed the following.

[2021-03-03 12:10:34,936] INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket)
[2021-03-03 12:10:34,941] INFO zookeeper.request.timeout value is 0. feature enabled= (org.apache.zookeeper.ClientCnxn)
[2021-03-03 12:10:34,944] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2021-03-03 12:10:34,947] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2021-03-03 12:10:34,952] INFO Socket connection established, initiating session, client: /127.0.0.1:40888, server: localhost/127.0.0.1:2181 (org.apache.zookeeper.ClientCnxn)
[2021-03-03 12:10:35,046] INFO Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x10000019ced0002, negotiated timeout = 18000 (org.apache.zookeeper.ClientCnxn)
[2021-03-03 12:10:35,052] INFO [ZooKeeperClient Kafka server] Connected. (kafka.zookeeper.ZooKeeperClient)
[2021-03-03 12:10:35,531] INFO Cluster ID = Tv2VkHIwQeeE9P8qlGcquQ (kafka.server.KafkaServer)
[2021-03-03 12:10:35,538] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.common.InconsistentClusterIdException: The Cluster ID Tv2VkHIwQeeE9P8qlGcquQ doesn't match stored clusterId Some(NojQoU95QE-ubFol4BNjXg) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong.
        at kafka.server.KafkaServer.startup(KafkaServer.scala:235)
        at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
        at kafka.Kafka$.main(Kafka.scala:82)
        at kafka.Kafka.main(Kafka.scala)
[2021-03-03 12:10:35,540] INFO shutting down (kafka.server.KafkaServer)
[2021-03-03 12:10:35,542] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
[2021-03-03 12:10:35,663] INFO Session: 0x10000019ced0002 closed (org.apache.zookeeper.ZooKeeper)
[2021-03-03 12:10:35,663] INFO EventThread shut down for session: 0x10000019ced0002 (org.apache.zookeeper.ClientCnxn)

So this line seems to hold the clue.

[2021-03-03 12:10:35,538] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.common.InconsistentClusterIdException: The Cluster ID Tv2VkHIwQeeE9P8qlGcquQ doesn't match stored clusterId Some(NojQoU95QE-ubFol4BNjXg) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong.
        at kafka.server.KafkaServer.startup(KafkaServer.scala:235)
        at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
        at kafka.Kafka$.main(Kafka.scala:82)
        at kafka.Kafka.main(Kafka.scala)

I may be wrong, completely wrong, please let me know. How to fix this "InconsistentClusterIdException" in kafka?

Answer 1

As I mentioned in the question my diagnosis revealed that "InconsistentClusterIdException" is the culprit in this situation. So somewhere a cluster.id is defined which is having an older value, and new cluster.id is not the same as the one given in the files defined in the system.

The log also mentions that the file where the cluster.id is defined is named "meta.properties", now the question is where to find this file.

I checked the kafka installation directory there is a logs folder, first I thought the meta.properties would be there but alas! that was not to be the case.

Then I ran the following command at the root directory to find the meta.properties file location.

kafka@bablo-HP-ProBook-440-G5:~$ sudo find / -type f -name meta.properties 
[sudo] password for kafka: 
find: ‘/run/user/1000/gvfs’: Permission denied
/var/lib/docker/volumes/5538f86a82c0c8e4b52b7c99d974f8d7f0dcd7cd5923372fc9526daa7610de77/_data/kafka-logs-0bd2b2fa548f/meta.properties
/var/lib/docker/volumes/5538f86a82c0c8e4b52b7c99d974f8d7f0dcd7cd5923372fc9526daa7610de77/_data/kafka-logs-2dbdd7463fde/meta.properties
/var/lib/docker/volumes/5538f86a82c0c8e4b52b7c99d974f8d7f0dcd7cd5923372fc9526daa7610de77/_data/kafka-logs-6adbc7f092cf/meta.properties
/var/lib/docker/volumes/5538f86a82c0c8e4b52b7c99d974f8d7f0dcd7cd5923372fc9526daa7610de77/_data/kafka-logs-f6f11dd5b1e3/meta.properties
/var/lib/docker/volumes/5538f86a82c0c8e4b52b7c99d974f8d7f0dcd7cd5923372fc9526daa7610de77/_data/kafka-logs-ecf97c2a08bd/meta.properties
/var/lib/docker/volumes/5538f86a82c0c8e4b52b7c99d974f8d7f0dcd7cd5923372fc9526daa7610de77/_data/kafka-logs-7bad187baa15/meta.properties
/var/lib/docker/volumes/5538f86a82c0c8e4b52b7c99d974f8d7f0dcd7cd5923372fc9526daa7610de77/_data/kafka-logs-e114b322d257/meta.properties
/var/lib/docker/volumes/5538f86a82c0c8e4b52b7c99d974f8d7f0dcd7cd5923372fc9526daa7610de77/_data/kafka-logs-de90efb99ab0/meta.properties
/var/lib/docker/volumes/5538f86a82c0c8e4b52b7c99d974f8d7f0dcd7cd5923372fc9526daa7610de77/_data/kafka-logs-5d1ae8504d91/meta.properties
/home/kafka/logs/meta.properties

Last line of the result mentioned the location of the meta.properties folder. It's located in /home/kafka/logs/meta.properties. At that location, I just commented the line containing the cluster.id key in meta.properties file and then saved it back.

Afterwards stopped the zookeeper and restarted both zookeeper and kafka.

That's it. It started again smoothly.