CODEWITHSUNDEEP
hadoopapache-kafkakafka-consumer-apikafka-producer-apibigdata
K S Nidhin
K S Nidhin

Reputation: 2650

Kafka console producer losing messages

I am using the below kafka console producer command to pass the contents of the file into the kafka producer.

sh ~/KAFKA_HOME/bin/kafka-console-producer.sh --broker-list xxx:9092,yyy:9092,zzz:9092 --topic HistLoad --new-producer < data.csv

The Data.csv file has about 700,000 records. I receive only about 699,800 messages at the consumer output.

I checked the offset counter for consumer and based on the offset values its having only 699,800 messages in the queue.

Could you help me in finidng out what is causing this issue of losing messages. What all do i need to check to get the root cause.

Upvotes: 3

Views: 954

Answers (1)

dawsaw
dawsaw

Reputation: 2313

This is because the console producer has acks=0 by default. Set request-required-acks to 1 and it should be fine.

For your reference https://issues.apache.org/jira/plugins/servlet/mobile#issue/KAFKA-3129

Upvotes: 2

Related Questions