CODEWITHSUNDEEP
javaapache-flink
jason zhang
jason zhang

Reputation: 451

Apache Flink: How are failures handled in the DataSet API?

I have two questions regarding the failure handling in Flink's DataSet API:

  1. Why isn't the checkpointing mechanism mentioned in documentation of the DataSet API?

  2. How are failures handled in the DataSet API, e.g., for reduce or reduceGroup transformation?

Upvotes: 0

Views: 140

Answers (1)

Fabian Hueske
Fabian Hueske

Reputation: 18987

Flink handles failures differently for streaming and batch programs.

For streaming programs, the input stream is unbound such that it is in general not possible or not feasible to replay the complete input in case of a failure. Instead Flink consistently checkpoints the state of operators and user functions and restores the state in case of a failure.

For batch programs, Flink recomputes intermediate results, which were lost due to failures, by reading the necessary input data and evaluating the relevant transformations again. This is true for all transformations, including reduce and reduceGroup.

Upvotes: 2

Related Questions