CODEWITHSUNDEEP
apache-kafka
maxenglander
maxenglander

Reputation: 4051

Kafka consumers logging "Auto offset commit failed" every few minutes

I am using the Kafka 0.9 "new" consumer API to connect to Kafka 0.9.0.0. Every couple of minutes, the following error is logged:

2016-01-27T16:38:14,584Z [pool-1-thread-1] ERROR o.a.k.c.c.i.ConsumerCoordinator - Auto offset commit failed. org.apache.kafka.clients.consumer.internals.SendFailedException: null

Here is a gist of my broker and consumer configs.

There are no corresponding/concerning log messages in the broker logs, and records appear to be flowing through my system just fine. This message is mainly an annoyance, causing distracting alerts and noise.

I'm curious to know what this exception means, but my real questions are:

  1. Should I be concerned by this exception?
  2. If I should not be concerned, is there a way to suppress it by tweaking the consumer/broker configs?
  3. If the answer to the two questions above is "no", what is a good rule for suppressing these messages (I can do it with Logback) so that I don't miss legitimate SendFailedException log messages

Upvotes: 2

Views: 9012

Answers (1)

Majid Hajibaba
Majid Hajibaba

Reputation: 3270

I have the same error message in different place (Heron and KafkaSpout)! My finding is the following:

If you use enable.auto.commit = true you must not use consumer.commitAsync();

Upvotes: 2

Related Questions