CODEWITHSUNDEEP
spring-bootspring-kafka
AJK1305
AJK1305

Reputation: 129

Spring Kafka Listener receiving duplicate messages

My Spring Kafka Listener is receiving duplicate messages, i can see the messages are being polled from same partition and offset and the same timestamp. In my code, i keep track every incoming message and identify the duplicates, but in this case, i cannot even reject it from processing as both messages - original and duplicate are received at almost same time, and the first record is not even committed in the database tracking table. 1.Please suggest how i should avoid polling the duplicate messages, i dont understand why it is being polled twice- only under load. 2. How i can handle this in tracking table, if message 1 metadata is being processed and not committed in

the tracking table, message 2 comes and is not able to find that record in tracking table and proceeds with processing the duplicate message again.

config of the listener based on my use case:
        config.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
        config.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, 300000);
        config.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 50); 
        config.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest");
        config.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 15000);

Upvotes: 2

Views: 2694

Answers (1)

Gary Russell
Gary Russell

Reputation: 174769

The two consumers need to have the same group.id property so that the partitions are distributed across them.

If they are in different consumer groups, they will both get all the records.

Upvotes: 1

Related Questions