CODEWITHSUNDEEP
apache-kafkakafka-consumer-api
tapu74
tapu74

Reputation: 1171

How to consume all messages from begining from apache kafka using java

I am trying to consume all messages from the beginning of a topic in Apache Kafka. I can consume messages which are producing at that moment. Here is my code for fetch messages.

 public void consume() {
    KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
    consumer.subscribe(Arrays.asList(topic));
    while (true) {
        ConsumerRecords<String, String> records = consumer.poll(1000);
        for (ConsumerRecord<String, String> record : records) {
            System.out.printf("\"%s\"\n", record.value());
        }
    }
}

Upvotes: 1

Views: 5780

Answers (3)

dossani
dossani

Reputation: 1948

Besides setting, auto.offset.reset=earliest, try setting a new/random value for property group.id and give it a try. Also, if you are not interested in keeping track of consumer position but always want to start from the beginning, you can also set enable.auto.commit=false to avoid polluting the offsets topic.

Hope it helps.

Thanks.

Upvotes: 2

OneCricketeer
OneCricketeer

Reputation: 191681

The console consumer generates a random consumer group id in order to accomplish this and sets auto.offset.reset=earliest

Make sure that you close the consumer object to prevent adding lots of temporary consumer groups ids in Zookeeper

Upvotes: 1

ppatierno
ppatierno

Reputation: 10065

After subscribing to the topic, you can use the seekToBeginning method in order to set the offset at the beginning of the topic partition. Of course, it's valid per partition because a topic with different partitions has different beginning offsets (if messages deletion happened).

Upvotes: 2

Related Questions