Reputation: 118
Does Apache Flink cache jobs?
I noticed that each time I run a new job it takes around 20% longer compared to the time when I launch it again?
Does flink cache some results and reuses them if a job is run multiple times? If so, how can I control this?
I would like to mesure how long my tasks run, but each time I rerun them it's faster than before.
Upvotes: 0
Views: 159
Answers (1)
Reputation: 680
If you using some stateful functions and configured checkpoints and savepoints your job can take a time to restore the state from checkpoints.
In order to make state fault tolerant, Flink needs to checkpoint the state. Checkpoints allow Flink to recover state and positions in the streams to give the application the same semantics as a failure-free execution.
More about checkpointing and here.
Upvotes: 1
Related Questions
- Does flink streaming job maintain its keyed value state between job runs?
- Does flink streaming have cache/persist feature? (like spark)
- What happens if Flink job restarts between two checkpoints?
- Is Flink job manager stateful or stateless?
- Flink runs multiple jobs from single jar
- Apache Flink - How Checkpoint/Savepoint works If we run duplicate jobs (Multi Tenancy)
- Apache Flink relating/caching data options
- Does flink reduce on the fly in batch mode
- Will Apache Flink restore trigger timers after failure?
- When are flink checkpoint files cleaned?