CODEWITHSUNDEEP
apache-flinkflink-streamingcheckpoint
Gaurav
Gaurav

Reputation: 173

Where is the state stored by default if I do not configure a StateBackend?

In my program I have enabled checkpointing,

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.enableCheckpointing(5000);

but I haven't configured any StateBackend.

Where is the checkpointed state stored? Can I somehow inspect this data?

Upvotes: 1

Views: 432

Answers (2)

axreldable
axreldable

Reputation: 164

It's not the answer, but small addition to the correct answer. I can't write comments cause of reputation.

If you use flink version earlier then v1.5 then default state backend will MemoryStateBeckend with asynchronous snapshots set to false. So you will use synchronous saving checkpoints every 5 seconds in your case (your pipeline will block every 5 seconds for saving checkpoint).

To avoid this use explicit constructor:

env.setStateBackend(new MemoryStateBackend(maxStateSize, true));

Since flink version v1.5.0 MemoryStateBackend uses asynchronous snapshots by default.

For more information see flink_v1.4 docs

Upvotes: 1

David Anderson
David Anderson

Reputation: 43419

The default state backend keeps the working state on the heaps of the various task managers, and backs that up to the job manager heap. This is the so-called MemoryStateBackend.

There's no API for directly accessing the data stored in the state backend. You can simulate a task manager failure and observe that the state is restored. And you can instead trigger a savepoint if you wish to externalize the state, though there is no tooling for directly inspecting these savepoints.

Upvotes: 1

Related Questions