CODEWITHSUNDEEP
javaapache-sparkapache-kafkaspark-streamingapache-zookeeper
Infamous
Infamous

Reputation: 754

Apache Kafka Cluster start fails with NoNodeException

I'm trying to start a spark streaming session which consumes from a Kafka queue and I'm using Zookeeper for config mgt. However, when I try to start this following exception is being thrown.

18/03/26 09:25:49 INFO ZookeeperConnection: Checking Kafka topic core-data-tickets does exists ...

18/03/26 09:25:49 INFO Broker: Kafka topic core-data-tickets exists
18/03/26 09:25:49 INFO Broker: Processing topic : core-data-tickets
18/03/26 09:25:49 WARN ZookeeperConnection: Resetting Topic Offset
org.I0Itec.zkclient.exception.ZkNoNodeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /consumers/clt/offsets/core-data-tickets/4
    at org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47)
    at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685)
    at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:766)
    at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:761)
    at kafka.utils.ZkUtils$.readData(ZkUtils.scala:443)
    at kafka.utils.ZkUtils.readData(ZkUtils.scala)
    at net.core.data.connection.ZookeeperConnection.readTopicPartitionOffset(ZookeeperConnection.java:145)

I have already created the relevant Kafka topic.

Any insights on this would be highly appreciated.

#

I'm using the following code to run the spark job

spark-submit --class net.core.data.compute.Broker     --executor-memory 512M     --total-executor-cores 2     --driver-java-options "-Dproperties.path=/ebs/tmp/continuous-loading-tool/continuous-loading-tool/src/main/resources/dev.properties"  --conf spark.ui.port=4045   /ebs/tmp/dev/data/continuous-loading-tool/target/continuous-loading-tool-1.0-SNAPSHOT.jar

Upvotes: 0

Views: 373

Answers (1)

Giorgos Myrianthous
Giorgos Myrianthous

Reputation: 39790

I guess that this error has to do with offsets retention. By default, offsets are stored for only 1440 minutes (i.e. 24 hours). Therefore, if the group has not committed offsets within a day, Kafka won't have information about it.

A possible workaround is to set the value of offsets.retention.minutes accordingly.

offsets.retention.minutes

Offsets older than this retention period will be discarded

Upvotes: 2

Related Questions