CODEWITHSUNDEEP
apache-flink
Shenjiaqi
Shenjiaqi

Reputation: 11

How does Flink save state in checkpoint/savepoint if some of state descriptor is removed

If I have a simple flink job with 2 keyed states, say State1 and State2.

The job is configured with rocksDB backend. Each of the states hold 10GB data.

If I update the code so that one of the state is not used(state descriptor deleted, and related code removed.). For example State1 is deleted.

When next time flink trigger checkpoint or I trigger savepoint manually. Will the checkpoint/savepoint still hold data of State1 or not?

Upvotes: 0

Views: 443

Answers (1)

David Anderson
David Anderson

Reputation: 43419

If you are using RocksDB with incremental checkpoints, then state for the obsolete state descriptor will remain in checkpoints until it is compacted away (but it can be ignored). With any full snapshot, nothing of State1 will remain.

With RocksDB, expired state is eventually removed by a RocksDB compaction filter. Until then, if StateTtlConfig.StateVisibility.NeverReturnExpired is set the state backend returns null in the place of expired values.

Documentation on State TTL

Upvotes: 1

Related Questions