CODEWITHSUNDEEP
javaapache-kafkaapache-zookeeperspring-kafkakafka-producer-api
mohsenJsh
mohsenJsh

Reputation: 2108

kafka does not have messages that producer sent

I use spring framework and kafka that has 3 brokers clustered. I found out that consumer did not consume some messages (let say 0.01 percent between all sent messages) so in producer code I log message offset that returned by api:

ListenableFuture<SendResult<String, Message>> future = messageTemplate.sendDefault(id, message);
SendResult<String, Message> sendResult = future.get();
String offset = sendResult.getRecordMetadata().offset();

I use return offset to query kafka topic in all partition but it do not find the message (I test other offsets related to messages that consumers used and they are in kafka), whats the problem and how can I insure that message sent to kafka??

I also used messageTemplate.flush(); in producer

Upvotes: 2

Views: 1206

Answers (1)

mohsenJsh
mohsenJsh

Reputation: 2108

I find out that when topic leader of Kafka broker goes down Kafka will rebalance itself and another broker becomes the leader of that partition and if the ack config is not set to all there is a chance of losing some data in this procedure. so change the config to 

ack=all

also, there is a chance of losing data if minimum in sync replica becomes less than 2, so set it at least to 2.

min.insync.replicas = 2

Upvotes: 1

Related Questions