CODEWITHSUNDEEP
apache-kafkakafka-producer-apiwal
user983447
user983447

Reputation: 1737

WAL in a Kafka producer

I have an application (in Java 8) that sends messages to Kafka. In order to improve the performance I want to configure the producer to send messages in batches (using ProducerConfig.BATCH_SIZE_CONFIG and ProducerConfig.LINGER_MS_CONFIG). But now, if I understand correctly, when for some reason the program crashes before sending the buffer, I will lose the messages.

In order to avoid this problem I would like to have to have some kind of WAL on the producer side. Is there any ready-made solution (program, library, Kafka configuration) that gives something like this? I would prefer not to implement it myself.

Upvotes: 0

Views: 243

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191983

There's no builtin solution for this, no. Therefore you'll need to implement this yourself, or add a shutdown hook to your application to flush the producer

Upvotes: 1

Related Questions