Reputation: 67
I have an IBM MQ input endpoint and a Kafka output endpoint in a camel route inside a sprint boot application. I want every message to land on Kafka exactly once. I have enabled idempotency and transactional id on kafka producer but in certain edge cases like MQ channel down abruptly I see duplicates in kafka topic. My theory is message on kafka is committed but mq channel goes down before its acknowledged to MQ and then when its back up, its replayed from source and sent to kafka again.
My question is that is there a way to handle this transactionally where wither both commits to kafka and mq happen or none. I cant use an external storage or anything for this. I understand that Kafka doesnt participate in XA transactions so my Atomikos transaction manager doesnt work in this case.
But I also see camel documentation that says
It would work with JTA camel-jta by using transacted() and if it involves some resources (SQL or JMS), which supports XA, then they would work in tandem, where they both will either commit or rollback at the end of the exchange routing. In some cases, if the JTA transaction manager fails to commit (during the 2PC processing), but kafka transaction has been committed before and there is no chance to roll back the changes since the kafka transaction does not support JTA/XA spec. There is still a risk with the data consistency.
So I am a bit confused whether it can be achieved or not. Any help is appreciated.
Upvotes: 0
Views: 33