Unable to list Kafka topics

Question

I am running ansible to install Kafka on one server. (on Dev env, standalone setup) but my playbook is getting failed for task listing of Kafka topics. below are the logs.

1. journalctl -fu zookeeper

-- Logs begin at Thu 2020-11-19 01:05:17 IST. --
Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:118)
Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:122)
Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:89)
Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:55)
Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:119)
Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
Nov 19 22:33:09 <HOSTNAME> systemd[1]: zookeeper.service: main process exited, code=exited, status=1/FAILURE
Nov 19 22:33:09 <HOSTNAME> systemd[1]: zookeeper.service: control process exited, code=exited status=1
Nov 19 22:33:09 <HOSTNAME> systemd[1]: Unit zookeeper.service entered failed state.
Nov 19 22:33:09 <HOSTNAME> systemd[1]: zookeeper.service failed.

journalctl -fu kafka.service
-- Logs begin at Thu 2020-11-19 01:05:17 IST. --
Nov 19 22:23:30 <HOSTNAME> systemd[1]: Started kafka.service.
Nov 19 22:23:40 <HOSTNAME> systemd[1]: kafka.service: main process exited, code=exited, status=1/FAILURE
Nov 19 22:23:40 <HOSTNAME> systemd[1]: kafka.service: control process exited, code=exited status=1
Nov 19 22:23:40 <HOSTNAME> systemd[1]: Unit kafka.service entered failed state.
Nov 19 22:23:40 <HOSTNAME> systemd[1]: kafka.service failed.
Nov 19 22:33:08 <HOSTNAME> systemd[1]: Started kafka.service.
Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service: main process exited, code=exited, status=1/FAILURE
Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service: control process exited, code=exited status=1
Nov 19 22:33:17 <HOSTNAME> systemd[1]: Unit kafka.service entered failed state.
Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service failed.

2. Getting same error when run command manually which playbook is trying to run.

/opt/kafka/kafka_2.12-2.2.2/bin/kafka-topics.sh --zookeeper <ip of localhost>:5181 --list

3. both Kafka and zookeeper are failed and not getting started.

  systemctl status  kafka.service -l
● kafka.service
   Loaded: loaded (/etc/systemd/system/kafka.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2020-11-19 22:33:17 IST; 5min ago
  Process: 5448 ExecStop=/opt/kafka/kafka_2.12-2.2.2/bin/kafka-server-stop.sh (code=exited, status=1/FAILURE)
  Process: 4742 ExecStart=/bin/sh -c /opt/kafka/kafka_2.12-2.2.2/bin/kafka-server-start.sh /opt/kafka/kafka_2.12-2.2.2/config/server.properties > /opt/kafka/kafka_2.12-2.2.2/kafka.log 2>&1 (code=exited, status=1/FAILURE)
 Main PID: 4742 (code=exited, status=1/FAILURE)

Nov 19 22:33:08 <HOSTNAME> systemd[1]: Started kafka.service.
Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service: main process exited, code=exited, status=1/FAILURE
Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service: control process exited, code=exited status=1
Nov 19 22:33:17 <HOSTNAME> systemd[1]: Unit kafka.service entered failed state.
Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service failed.


 systemctl status  zookeeper.service  -l
● zookeeper.service
   Loaded: loaded (/etc/systemd/system/zookeeper.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2020-11-19 22:33:09 IST; 5min ago
  Process: 5281 ExecStop=/opt/kafka/kafka_2.12-2.2.2/bin/zookeeper-server-stop.sh (code=exited, status=1/FAILURE)
  Process: 4448 ExecStart=/opt/kafka/kafka_2.12-2.2.2/bin/zookeeper-server-start.sh /opt/kafka/kafka_2.12-2.2.2/config/zookeeper.properties (code=exited, status=1/FAILURE)
 Main PID: 4448 (code=exited, status=1/FAILURE)

Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:118)
Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:122)
Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:89)
Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:55)
Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:119)
Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
Nov 19 22:33:09 systemd[1]: zookeeper.service: main process exited, code=exited, status=1/FAILURE
Nov 19 22:33:09  systemd[1]: zookeeper.service: control process exited, code=exited status=1
Nov 19 22:33:09  systemd[1]: Unit zookeeper.service entered failed state.
Nov 19 22:33:09  systemd[1]: zookeeper.service failed.

4. Kafka.log

[2020-11-19 22:33:09,853] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2020-11-19 22:33:10,584] INFO starting (kafka.server.KafkaServer)
[2020-11-19 22:33:10,585] INFO Connecting to zookeeper on <ip of localhost>:5181 (kafka.server.KafkaServer)
[2020-11-19 22:33:10,640] INFO [ZooKeeperClient] Initializing a new session to <ip of localhost>:5181. (kafka.zookeeper.ZooKeeperClient)
[2020-11-19 22:33:10,646] INFO Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,646] INFO Client environment:host.name=<FQDN> (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,646] INFO Client environment:java.version=1.8.0_221 (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,646] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,646] INFO Client environment:java.home=/opt/jre1.8.0_221 (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,646] INFO Client environment:java.class.path=/opt/kafka/kafka_2.12-2.2.2/bin/../libs/activation-1.1.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/aopalliance-repackaged-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/audience-annotations-0.5.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/commons-lang3-3.8.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-api-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-basic-auth-extension-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-file-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-json-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-runtime-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-transforms-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/guava-20.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/hk2-api-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/hk2-locator-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/hk2-utils-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-annotations-1.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.inject-1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.inject-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.ws.rs-api-2.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-client-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-common-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-container-servlet-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-container-servlet-core-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-hk2-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-media-jaxb-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-server-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-client-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-continuation-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-streams-examples-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-streams-scala_2.12-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-streams-test-utils-2.2.2.jar:/opt/kafka/kafka_2.12-2-2.2.2/bin/../libs/validation-api-1.1.0.Final.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zkclient-0.11.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zookeeper-3.4.13.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zstd-jni-1.3.8-1.jar (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,646] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,646] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,647] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,647] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,647] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,647] INFO Client environment:os.version=3.10.0-1062.el7.x86_64 (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,647] INFO Client environment:user.name=kafka (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,647] INFO Client environment:user.home=/home/kafka (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,647] INFO Client environment:user.dir=/ (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,648] INFO Initiating client connection, connectString=<ip of localhost>:5181 sessionTimeout=6000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@3c419631 (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,667] INFO Opening socket connection to server <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:10,673] INFO Socket error occurred: <FQDN>/<ip of localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:10,676] INFO [ZooKeeperClient] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2020-11-19 22:33:11,796] INFO Opening socket connection to server <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:11,797] INFO Socket error occurred: <FQDN>/<ip of localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:12,897] INFO Opening socket connection to server <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:12,898] INFO Socket error occurred: <FQDN>/<ip of localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:13,999] INFO Opening socket connection to server <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:13,999] INFO Socket error occurred: <FQDN>/<ip of localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:15,101] INFO Opening socket connection to server <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:15,101] INFO Socket error occurred: <FQDN>/<ip of localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:16,202] INFO Opening socket connection to server <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:16,202] INFO Socket error occurred: <FQDN>/<ip of localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:16,678] INFO [ZooKeeperClient] Closing. (kafka.zookeeper.ZooKeeperClient)
[2020-11-19 22:33:17,303] INFO Opening socket connection to server <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:17,406] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:17,407] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:17,409] INFO [ZooKeeperClient] Closed. (kafka.zookeeper.ZooKeeperClient)
[2020-11-19 22:33:17,412] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
        at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:242)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
        at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:238)
        at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:96)
        at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1825)
        at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:361)
        at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:385)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:205)
        at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
        at kafka.Kafka$.main(Kafka.scala:75)
        at kafka.Kafka.main(Kafka.scala)
[2020-11-19 22:33:17,415] INFO shutting down (kafka.server.KafkaServer)
[2020-11-19 22:33:17,422] INFO shut down completed (kafka.server.KafkaServer)
[2020-11-19 22:33:17,422] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
[2020-11-19 22:33:17,426] INFO shutting down (kafka.server.KafkaServer)

5. /var/log/messages

Nov 19 18:16:07 <HOSTNAME> ansible-command: Invoked with creates=None executable=None _uses_shell=False strip_empty_ends=True _raw_params=/opt/kafka/kafka_
2.12-2.2.2/bin/kafka-topics.sh --zookeeper <FQDN>:5181 --list removes=None argv=None warn=True chdir=None stdin_add_newline=True stdin=None
Nov 19 18:16:07 <HOSTNAME> ansible-command: Invoked with creates=None executable=None _uses_shell=False strip_empty_ends=True _raw_params=/opt/kafka/kafka_
2.12-2.2.2/bin/kafka-topics.sh --zookeeper <FQDN>:5181 --list removes=None argv=None warn=True chdir=None stdin_add_newline=True stdin=None
Nov 19 18:16:07 <HOSTNAME> ansible-command: Invoked with creates=None executable=None _uses_shell=False strip_empty_ends=True _raw_params=/opt/kafka/kafka_
2.12-2.2.2/bin/kafka-topics.sh --zookeeper <FQDN>:5181 --list removes=None argv=None warn=True chdir=None stdin_add_newline=True stdin=None
Nov 19 18:16:07 <HOSTNAME> heartbeat: 2020-11-19T18:16:07.941+0530#011INFO#011kafka/log.go:53#011client/metadata fetching metadata for [heartbeat] from bro
ker <ip of localhost>:3997
Nov 19 18:16:07 <HOSTNAME> heartbeat: 2020-11-19T18:16:07.942+0530#011INFO#011kafka/log.go:53#011Failed to connect to broker <ip of localhost>:3997: dial tcp
<ip of localhost>:3997: connect: connection refused
Nov 19 18:16:07 <HOSTNAME> heartbeat: 2020-11-19T18:16:07.942+0530#011INFO#011kafka/log.go:53#011client/metadata got error from broker -1 while fetching me
tadata: dial tcp <ip of localhost>:3997: connect: connection refused
Nov 19 18:16:07 <HOSTNAME> heartbeat: 2020-11-19T18:16:07.942+0530#011INFO#011kafka/log.go:53#011kafka message: client/metadata no available broker to send
 metadata request to
Nov 19 18:16:07 <HOSTNAME> heartbeat: 2020-11-19T18:16:07.942+0530#011INFO#011kafka/log.go:53#011client/brokers resurrecting 1 dead seed brokers
Nov 19 18:16:07 <HOSTNAME> heartbeat: 2020-11-19T18:16:07.942+0530#011INFO#011[publisher]#011pipeline/retry.go:155#011Drop batch
Nov 19 18:16:07 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:07,962] INFO Reading configuration from: /opt/kafka/kafka_2.12-2.2.2/config/zookeep
er.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,009] INFO Resolved hostname: <FQDN> to address: <FQDN>/<ip of localhost> (org.apache.zookeeper.server.quorum.QuorumPeer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,009] ERROR Invalid configuration, only one server specified (ignoring) (org.apac
he.zookeeper.server.quorum.QuorumPeerConfig)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,011] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.Datadi
rCleanupManager)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,011] INFO autopurge.purgeInterval set to 0 (org.apache.zookeeper.server.DatadirC
leanupManager)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,011] INFO Purge task is not scheduled. (org.apache.zookeeper.server.DatadirCleanupManager)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,011] WARN Either no config or no quorum defined in config, running  in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain

Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,027] INFO Reading configuration from: /opt/kafka/kafka_2.12-2.2.2/config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,027] INFO Resolved hostname: <FQDN> to address: <FQDN>/<ip of localhost> (org.apache.zookeeper.server.quorum.QuorumPeer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,027] ERROR Invalid configuration, only one server specified (ignoring) (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,027] INFO Starting server (org.apache.zookeeper.server.ZooKeeperServerMain)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,033] INFO Server environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,033] INFO Server environment:host.name=<FQDN> (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,033] INFO Server environment:java.version=1.8.0_221 (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,033] INFO Server environment:java.vendor=Oracle Corporation (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,033] INFO Server environment:java.home=/opt/jre1.8.0_221 (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,033] INFO Server environment:java.class.path=/opt/kafka/kafka_2.12-2.2.2/bin/../libs/activation-1.1.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/aopalliance-repackaged-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/argparse4j-extension-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-transforms-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-core-CR2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.annotation-api-1.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.inject-1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.inject-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.ws.rs-api-2.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-client-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-common-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-container-servlet-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-container-servlet-core-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-client-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-continuation-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-streams-scala_2.12-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-streams-test-utils-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/osgi-resource-locator-1.0.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/plexus-utils-3.1.0.jar:/opt/kafka/kafka_2.12-1.1.7.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/validation-api-1.1.0.Final.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zkclient-3.4.13.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zstd-jni-1.3.8-1.jar (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034] INFO Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034] INFO Server environment:java.io.tmpdir=/tmp (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034] INFO Server environment:java.compiler=<NA> (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034] INFO Server environment:os.name=Linux (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034] INFO Server environment:os.arch=amd64 (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034] INFO Server environment:os.version=3.10.0-1062.el7.x86_64 (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034] INFO Server environment:user.name=kafka (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034] INFO Server environment:user.home=/home/kafka (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034] INFO Server environment:user.dir=/ (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,043] INFO tickTime set to 3000 (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,043] INFO minSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,044] INFO maxSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,049] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,054] INFO binding to port 0.0.0.0/0.0.0.0:5181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,131] ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.ZooKeeperServerMain)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: java.io.EOFException
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at java.io.DataInputStream.readInt(DataInputStream.java:392)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.persistence.FileHeader.deserialize(FileHeader.java:66)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCreated(FileTxnLog.java:585)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createInputArchive(FileTxnLog.java:604)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNextLog(FileTxnLog.java:570)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:650)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:219)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:176)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:217)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:284)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:407)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:118)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:122)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:89)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:55)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:119)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
Nov 19 18:16:08 <HOSTNAME> systemd: zookeeper.service: main process exited, code=exited, status=1/FAILURE
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-stop.sh: No zookeeper server to stop
Nov 19 18:16:08 <HOSTNAME> systemd: zookeeper.service: control process exited, code=exited status=1
Nov 19 18:16:08 <HOSTNAME> systemd: Unit zookeeper.service entered failed state.
Nov 19 18:16:08 <HOSTNAME> systemd: zookeeper.service failed.

6. /opt/kafka/kafka_2.12-2.2.2/config/server.properties

broker.id=1
advertised.listeners=PLAINTEXT://hostname of localhost:3997
listeners=PLAINTEXT://hostname of localhost:3997

delete.topic.enable=true
auto.create.topics.enable=True
offsets.topic.replication.factor=1

# The number of threads handling network requests
num.network.threads=3

# The number of threads doing disk I/O
num.io.threads=6

# The send buffer (SO_SNDBUF) used by the socket server
#socket.send.buffer.bytes=102400

# The receive buffer (SO_RCVBUF) used by the socket server
#socket.receive.buffer.bytes=102400

# The maximum size of a request that the socket server will accept (protection against OOM)
#socket.request.max.bytes=104857600

# A comma seperated list of directories under which to store log files
log.dirs=/var/log/kafka

# The default number of log partitions per topic.
num.partitions=3

# Default Replication Factor
default.replication.factor=1

# The minimum age of a log file to be eligible for deletion
log.retention.hours=168

# A size-based retention policy for logs. Segments are pruned from the log as long as the remaining
# segments don't drop below log.retention.bytes.
#log.retention.bytes=1073741824

# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=1073741824

# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms=300000

zookeeper.connect= <ip of localhost>:5181
zookeeper.connection.timeout.ms=6000

/opt/kafka/kafka_2.12-2.2.2/config/zookeeper.properties

# clientPortBindAddress=localhost
clientPort=5181

dataDir=/var/data/zookeeper
dataLogDir=/var/data/logs/zookeeper
maxClientCnxns=0

# time for inital synchronization
initLimit=10
# how many ticks can pass before timeout
syncLimit=5

server.1=<hostname of localhost>:2888:3888

zookeeper.service

[Unit]
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
User=kafka
ExecStart=/opt/kafka/kafka_2.12-2.2.2/bin/zookeeper-server-start.sh /opt/kafka/kafka_2.12-2.2.2/config/zookeeper.properties
ExecStop=/opt/kafka/kafka_2.12-2.2.2/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Answer 1

all four words commands (stat,ruok) where giving connection refused.

I moved all files (log.1,log.103,log.142,log.16a,log.17d - all where having oct timestamp) present inside /var/data/logs/zookeeper/version-2/ to some other directory and started zookeeper and then kafka and both started and are up and running. I am able to list the topics now. now I am see log.1 file got created. (its binary so can't see inside it)

Can any one know the reason behind it? why it worked after moving these log* files which seems to be little old.

not sure if same has to do after 1 month from now if same issue occurs again.