CODEWITHSUNDEEP
javaapache-flink
jiemar leo
jiemar leo

Reputation: 61

flink broadcast state doesn't save to rocksdb when checkpoint happened?

No RocksDB state backend: Broadcast state is kept in-memory at runtime and memory provisioning should be done accordingly. This holds for all operator states.

above is from flink official website. does it mean broadcast state will not be stored at rocksdb? even when checkpoint happened?

Upvotes: 2

Views: 710

Answers (1)

David Anderson
David Anderson

Reputation: 43707

Not to worry, broadcast state (like all operator state) is included in Flink's checkpoints. But its working state is in memory (on the JVM heap) regardless of the choice of state backend. Only keyed state has the option of being stored in RocksDB.

RocksDB is a local, embedded key/value store that keeps its working state on the local disk, with an off-heap cache. If you are using RocksDB as your state backend, then when checkpoints occur both the on-heap operator state and the keyed state from RocksDB will be copied into the checkpoint. The checkpoints are not stored in RockDB, but are instead written to an off-node distributed filesystem for durability.

Upvotes: 3

Related Questions