CODEWITHSUNDEEP
apache-flinkflink-streaming
Jobs
Jobs

Reputation: 1307

Flink Savepoint data folder is missing

I was testing Flink savepoint locally.

While taking savepoint, I mentioned the folder, and save point has been taken into it.

Later I restarted the Flink cluster and restored the Flink job from the savepoint and it worked as expected.

My concern is about the save point folder contents.

I am only seeing the _metadata file in the folder.

Where will the savepoint data get saved?

It is clear from the documentation that( https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/savepoints/#triggering-savepoints )

"If you use statebackend: jobmanager, metadata and savepoint state will be stored in the _metadata file, so don’t be confused by the absence of additional data files."

But I have used rocksDb backend for states.

state.backend: rocksdb
state.backend.incremental: true
state.backend.rocksdb.ttl.compaction.filter.enabled: true

enter image description here

Thanks for the helps in advance.

Upvotes: 2

Views: 414

Answers (1)

David Anderson
David Anderson

Reputation: 43439

If your state is small, it will be stored in the metadata file. The definition of "small" is controlled by state.backend.fs.memory-threshold. See https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/config/#state-storage-fs-memory-threshold for more info.

This is done this way to prevent creating lots of small files, which causes problems with some filesystems, e.g., S3.

Upvotes: 4

Related Questions