CODEWITHSUNDEEP
apache-kafkakafka-consumer-api
Aravind Yarram
Aravind Yarram

Reputation: 80186

Kafka consumer 100% cpu usage

I have a kafka consumer which has been running for few days perfectly. Yesterday the broker went down. Since then the consumer is showing 100% cpu usage. The usage went down considerably (under 10%) when the broker was up. I am curious why the usage is 100% (or near 100%) when broker is down? I don't see anything in the logs.

Upvotes: 5

Views: 11665

Answers (2)

Jai Gupta
Jai Gupta

Reputation: 31

Increase in CPU utilisation for both kafka consumer and broker usually happens because of very high frequency of poll requests from consumer. This mainly happens because of inefficient configuration for consumer.

Check for two properties :

  1. fetch.min.bytes -> Minimum bytes to finish poll request
  2. fetch.max.wait.ms -> Max wait at broker to finish poll request

If data in topic is not that frequently produced then keep fairly high value of fetch.max.wait.ms and non-zero value of fetch.min.bytes to avoid frequent poll requests to broker

Upvotes: 3

Stepan Yakovenko
Stepan Yakovenko

Reputation: 9206

I've solved the problem by upgrading kafka from:

    <dependency>
        <groupId>org.apache.kafka</groupId>            
        <artifactId>kafka_2.11</artifactId>
        <version>0.11.0.1</version>
    </dependency>

to

    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka_2.12</artifactId>
        <version>2.0.1</version>
    </dependency>

Upvotes: 1

Related Questions