CODEWITHSUNDEEP
apache-kafkaspring-transactionsspring-kafka
Rahul Gupta
Rahul Gupta

Reputation: 1139

Exactly-once semantics in spring Kafka

I need to apply transactions in a system that comprises of below components:

  1. A Kafka producer, this is some external application which would publish messages on a kafka topic.
  2. A Kafka consumer, this is a spring boot application where I have configured the kafka listener and after processing the message, it needs to be saved to a NoSQL database.

I have gone through several blogs like this & this, and all of them talks about the transactions in context of streaming application, where the messages would be read-processed-written back to a Kafka topic.

I don't see any clear example or blog around achieving transactionality in the use case similar to mine i.e. producing-processing-writing to a DB in a single atomic transaction. I believe it to be very common scenario & there must be some support for it as well.

Can someone please guide me on how to achieve this? Any relevant code snippet would be greatly appreciated.

Upvotes: 4

Views: 980

Answers (2)

b15
b15

Reputation: 2361

I'm guessing you're trying to solve the scenario where your consumer goes down after writing to the database but before committing the offsets, or other similar problems. Unfortunately this means you have to build your own fault-tolerance.

In the case of the problem I mentioned above, this means you would have to manage the consumer offsets in your end-output database, updating them in the same database transaction that you're writing the output of your consumer application to.

Upvotes: 1

Gary Russell
Gary Russell

Reputation: 174769

in a single atomic transaction.

There is no way to do it; Kafka doesn't support XA transactions (nor do most NoSQL DBs). You can use Spring's transaction synchronization for best-effort 1PC.

See the documentation.

Spring for Apache Kafka implements normal Spring transaction synchronization.

It provides "best efforts 1PC" - see Distributed transactions in Spring, with and without XA for more understanding and the limitations.

Upvotes: 1

Related Questions