CODEWITHSUNDEEP
javaapache-kafkakafka-consumer-api
tsar2512
tsar2512

Reputation: 2994

Kafka Consumer Properties to read from the maximum offset

I have written a Java Kafka Consumer. I would like to make sure how to explicitly ensure that once the Kafka Consumer is started it only reads the messages which are sent by the producer from that time onwards i.e. it should not read any messages which have already been sent by the producer to Kafka. Can anyone explain how to ensure this? :

Here is a snippet of the properties I use

Properties properties = new Properties();
properties.put("zookeeper.connect", zookeeperHost);
properties.put("group.id", group);
properties.put("auto.offset.reset","largest");
ConsumerConfig consumerConfig = new ConsumerConfig(properties);
consumerConnector = Consumer.createJavaConsumerConnector(consumerConfig);

UPDATE Sept14:

I am using the following properties, it seems that the consumer still reads from the beginning at times, can someone tell me what's wrong now?

I am using Kafka Version 0.8.2

properties.put("auto.offset.reset","largest");
properties.put("auto.commit.enable","false");

Upvotes: 2

Views: 2402

Answers (1)

tsar2512
tsar2512

Reputation: 2994

Based on answers above, it seems that the correct mechanism is as follows for setting properties of the consumer:

 properties.put("auto.offset.reset","largest");
 properties.put("auto.commit.enable","false");

This ensures reading from the maximum offset

Upvotes: 2

Related Questions