CODEWITHSUNDEEP
apache-kafkakafka-consumer-api
Joey Trang
Joey Trang

Reputation: 1173

Problems with the retention period for offset topic of kafka

We are facing a problem with Kafka. We have a topic with only a partition and only one consumer in a consumer group. The consumer has been stopped for a month. In the meantime, producers are sending messages to the topic.

When we start the consumer again, it is not able to consume messages. I assume that the previously committed offset has been lost, so the consumer has no idea to find the starting point when awaken.

When we stop and start the consumer again, then the consumer can pick up the new messages, but all message that has been sent previously never got consumed.

Has offset been corrupted? Does the retention period for the kafka internal topic offsets, mean that the last committed offsets been has removed?

Upvotes: 12

Views: 25488

Answers (3)

Ashok Kumar Sahoo
Ashok Kumar Sahoo

Reputation: 580

The retention value can be configured in kafka broker using:

offsets.retention.minutes

The default is 24 hours. See: Kafka consumer group offset retention

Edit: As of Kafka 2.0, the default value is 7 days.

Upvotes: 10

Naresh Kumar Kotha
Naresh Kumar Kotha

Reputation: 39

we were also facing the same issue and I have researched a bit about it. I have the changed below steps it resolved .

We have retention of 14 days so we changed the kafka offsets topic retention to 14 days

We have changed cleanup.policy from compact to delete by running below 

./kafka-configs.sh  --alter --zookeeper localhost:2181 --entity-type topics
              --entity-name __consumer_offsets --add-config cleanup.policy=delete

Updated config for topic: "__consumer_offsets".

Upvotes: 3

Mani Maran
Mani Maran

Reputation: 11

The default retention period for the message(offset) in kafka is a week i.e)7 days

After the retention period all the existing offsets will be deleted from the broker due to avoid overflow of the space.

so the previous committed offset will be changed to latest offset as the previous offset has been removed by kafka / zookeeper.

If you want to have your message for long period, Specify --retention-period value while creating topic in kafka.

Upvotes: 1

Related Questions