CODEWITHSUNDEEP
apache-flinkrocksdbcheckpoint
Jerry Zhang
Jerry Zhang

Reputation: 247

How does Flink make checkpoint asynchronously with RocksDB backend

I am using Flink with RocksDB. From the document of Flink I acknowledge that Flink will make checkpoint asynchronously when using RocksDB backend. See the descriptions in its doc.

It is possible to let an operator continue processing while it stores its state snapshot, effectively letting the state snapshots happen asynchronously in the background. To do that, the operator must be able to produce a state object that should be stored in a way such that further modifications to the operator state do not affect that state object. For example, copy-on-write data structures, such as are used in RocksDB, have this behavior.

From my understanding, when a checkpoint need to be make, an operator will do these steps for Rocksdb:

  1. Flush data in memtable
  2. Copy the db folder into another tmp folder, which contains all the data in RocksDB
  3. Upload the copied data to remote Fs-system. (In this step, it is asynchronous)

Is my understanding right ? Or could anyone help to illustrate the details ?

Thanks a lot because I cannot find any documentation to describe the details.

Upvotes: 3

Views: 1199

Answers (1)

Jerry Zhang
Jerry Zhang

Reputation: 247

Found one Blog where mentioned the process:

To do this, Flink triggers a flush in RocksDB, forcing all memtables into sstables on disk, and hard-linked in a local temporary directory. This process is synchronous to the processing pipeline, and Flink performs all further steps asynchronously and does not block processing.

See the link for more details: Incremental Checkpoint

Upvotes: 3

Related Questions