CODEWITHSUNDEEP
hadoopapache-kafkahadoop2rhelcloudera-cdh
BruceWayne
BruceWayne

Reputation: 3374

Kafka configuration on existing CDH 5.5.2 cluster

I am installing Kafka-2.0 on my existing CDH 5.5.2 cluster, here is the procedure what i followed

  1. Add services from CM
  2. Selected Kafka (Before that i downloaded and distributes and activated kafka parcel on all the nodes)
  3. Selected 1 nodes for KafkaBroker and 4 nodes for Kafka MirrorMaker
  4. Then i updated my Destination Broker List (bootstrap.servers) property with one of the Mirror Maker node as well as Source Broker List (source.bootstrap.servers) with same node

  5. Below error i am getting (log file)

    Fatal error during KafkaServerStartable startup. Prepare to shutdown 
java.lang.OutOfMemoryError: Java heap space
    at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
    at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
    at kafka.log.SkimpyOffsetMap.<init>(OffsetMap.scala:43)
    at kafka.log.LogCleaner$CleanerThread.<init>(LogCleaner.scala:186)
    at kafka.log.LogCleaner$$anonfun$1.apply(LogCleaner.scala:83)
    at kafka.log.LogCleaner$$anonfun$1.apply(LogCleaner.scala:83)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
    at scala.collection.immutable.Range.foreach(Range.scala:166)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
    at scala.collection.AbstractTraversable.map(Traversable.scala:104)
    at kafka.log.LogCleaner.<init>(LogCleaner.scala:83)
    at kafka.log.LogManager.<init>(LogManager.scala:64)
    at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:601)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:180)
    at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
    at kafka.Kafka$.main(Kafka.scala:67)
    at com.cloudera.kafka.wrap.Kafka$.main(Kafka.scala:76)
    at com.cloudera.kafka.wrap.Kafka.main(Kafka.scala)

Upvotes: 0

Views: 2315

Answers (2)

Prasad Khode
Prasad Khode

Reputation: 6739

you need to increase broker_max_heap_size value to atleast 1GB and restart the kafka service from Cloudera Manager. If you still face the same issue, try to increase as per your cluster configurations

enter image description here

Upvotes: 5

Roshith
Roshith

Reputation: 2175

The stack trace shows "java.lang.OutOfMemoryError: Java heap space" - VM heap is running out of space. Increase it by setting 

export KAFKA_HEAP_OPTS="-Xmx1G -Xms512M"

in /bin/kafka-server-start.sh.

Upvotes: 1

Related Questions