CODEWITHSUNDEEP
transactionsapache-kafkakafka-producer-apiatomic
rh0x
rh0x

Reputation: 1463

Guarantee unique global transaction for Kafka Producers

With the last version of Kafka 0.11.0.0 the Apache team is introducing idempotent producer and transactions. Is It possible to guarantee that an entire set of messages (for example 1 million) we want to log, will be committed only at the end? I would like that, if for example the Producers loose the connection with the brokers and cannot restabilish it, no messages will be seen by the consumers. Is it possible?

Upvotes: 1

Views: 3445

Answers (1)

Hans Jespersen
Hans Jespersen

Reputation: 8335

Yes this is possible using Transactions in your producer. You start a transaction, publish all your messages, and then commit the transaction. All the messages are written to Kafka one at a time but consumers in the new READ_COMMITTED mode will only see the messages after the transaction is committed by the producer and a special transaction marker is added to the Kafka commit log.

Consumers not in READ_COMMITTED mode can see the messages as they are written individually even though they may not yet (or ever) be committed.

There is a limit to how long an open transaction can stay uncommitted so eventually if the producer dies and does not explicitly end the transaction it will timeout and rollback and READ_COMMITTED consumers will never see those messages.

Upvotes: 7

Related Questions