CODEWITHSUNDEEP
apache-kafka
DJ180
DJ180

Reputation: 19854

Kafka 0.9: Consume from earliest Kafka offset

From my reading of the Kafka Consumer configs my understanding was that if I set these 2 properties, then on launch of my Consumer I will always consume from the earliest offset

enable.auto.commit = false
auto.offset.reset = earliest

While this works when I start my application for the first time, the next time I restart it does not consume from the beginning

Instead, what I need to do is change my group.id to something new and then it will resume from the earliest offset.

Could there possibly be some other committing going on?

Update

Looks to me like this is a problem with the Camel Kafka component that I am using. The org.apache.camel.component.kafka.KafkaConsumer class has this logic

if (endpoint.isAutoCommitEnable() != null && !endpoint.isAutoCommitEnable()) {
    if (processed >= endpoint.getBatchSize()) {
        consumer.commitSync();
        processed = 0;
    }
}

by my reading this looks like every time auto commit enable is false it will commit the offset. This is a feature of the Camel Kafka component, in that it will synch after x number of messages even if auto commit is enabled

Upvotes: 1

Views: 3763

Answers (1)

Chris Sainty
Chris Sainty

Reputation: 9346

Your understanding sounds correct.

Kafka 0.9 has both "Old" and "New" consumer configs. This configuration property changed between them.

auto.commit.enable = false
enable.auto.commit = false

https://kafka.apache.org/documentation#consumerconfigs

On startup it should be logging its configuration as well, so verify there.

2016-10-06 14:19:41,725 INFO [org.apache.kafka.clients.consumer.ConsumerConfig:165] - ConsumerConfig values:
    group.id = service
    bootstrap.servers = [kafka:9092]
    enable.auto.commit = false
    auto.offset.reset = latest

Upvotes: 3

Related Questions