CODEWITHSUNDEEP
apache-flinkflink-streaming
Derek X.
Derek X.

Reputation: 11

Details: How Flink achieves exactly-once mechanism?

From a previous post, it seems Flink achieves exactly-once by

After a successful pre-commit, the commit must be guaranteed to eventually succeed

I think "a successful pre-commit" is achieved by Flink Task Manager; and the "eventual succeed" is achieved by the Flink sink.

  1. How Flink sink node achieves the "eventual succeed"?
  2. Does this exactly-once mechanism have anything to do with checkpoint?

Upvotes: 1

Views: 165

Answers (1)

David Anderson
David Anderson

Reputation: 43454

Flink's two-phase commit sinks typically couple their actions with the checkpointing mechanism in the following way:

  • onSnapshot: Flush all records and pre-commit
  • onCheckpointComplete: Commit pending transactions and publish data
  • onRecovery: Check and commit any pending transactions

Note that it is possible for data to be lost if the external system times out pending transactions that would be committed during the onRecovery phase.

You can learn more about this in An Overview of End-to-End Exactly-Once Processing in Apache Flink (with Apache Kafka, too!).

Upvotes: 2

Related Questions