CODEWITHSUNDEEP
pythonapache-kafka
Tampa
Tampa

Reputation: 78254

python and kafka KeyError: -1

Python kafka does not work. Any reason why? I mean I cant event connect to kafka and i get an error? Is there a kafka client that works with 0.8? I mean this is a brand new server. I just booted it up.
I am using https://github.com/mumrah/kafka-python 

from kafka.client import KafkaClient
kafka = KafkaClient(kafka_domain, 9092)


Traceback (most recent call last):
  File "/home/ubuntu/workspace/rtbhui-devops/simulations/pixel_druid_simulations.py", line 36, in <module>
    kafka = KafkaClient(kafka_domain, 9092)
  File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line 38, in __init__
    self.load_metadata_for_topics()  # bootstrap with all metadata
  File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line 247, in load_metadata_for_topics
    self.topics_to_brokers[topic_part] = brokers[meta.leader]
KeyError: -1

In kafka logs I see the below.

[2014-02-26 08:36:21,471] INFO Closing socket connection to /222.127.xxx.xxx. (kafka.network.Processor)
[2014-02-26 08:40:30,801] ERROR [KafkaApi-1393401480] Error while fetching metadata for partition [topic-pixel,0] (kafka.server.KafkaApis)
kafka.common.LeaderNotAvailableException: Leader not available for partition [topic-pixel,0]
    at kafka.server.KafkaApis$$anonfun$17$$anonfun$20.apply(KafkaApis.scala:474)
    at kafka.server.KafkaApis$$anonfun$17$$anonfun$20.apply(KafkaApis.scala:462)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
    at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
    at scala.collection.immutable.List.foreach(List.scala:45)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
    at scala.collection.immutable.List.map(List.scala:45)
    at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:462)
    at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:458)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
    at scala.collection.immutable.Set$Set1.foreach(Set.scala:81)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
    at scala.collection.immutable.Set$Set1.map(Set.scala:68)
    at kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:458)
    at kafka.server.KafkaApis.handle(KafkaApis.scala:68)
    at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42)
    at java.lang.Thread.run(Thread.java:744)

Upvotes: 1

Views: 1932

Answers (1)

dpkp
dpkp

Reputation: 1459

This is fixed in kafka-python version 0.9.0. The error is internal to Kafka Python, not the kafka server, and simply involves handling partitions that are currently without a leader (which happens for any new topic when you auto-create topics, but other than that is fairly rare under normal operations)

See

https://github.com/mumrah/kafka-python/pull/109

Upvotes: 1

Related Questions