CODEWITHSUNDEEP
apache-flinkflink-streaming
yaarix
yaarix

Reputation: 500

Flink checkpoints interval and state size

We are running a few flink jobs, all of which have a kafka source and multiple cassandra sinks. We are heavily relying on time windows with reduce function, on keyed data. Our TPS is currently around 100—200.

I have a few questions about checkpoints and the size of the state that being saved:

  1. Since we're using reduce function, is the state size only influenced by the number of opened windows? If an hourly window and a minute window both have same accumaltor, should we expect a similar state size? For some reason were seeing that hourly window has much larger state than minute window, and daily window has larger state than hourly window.

  2. What is considered to be a reasonable amount of opened windows? What is considered to be a large state? What are the most common checkpoint time intervals (ours is 5 seconds which seems far too often to me), how long should we expect a checkpoint save time to take in a reasonable storage, for 1 gb of state? How TBs of state (which i read some system has) can be checkpointed in a reasonable amount of time? I know these are abstract questions but were not sure that our flink setup is working as expected and what to expect as our data grows.

  3. Were seeing both async and sync checkpoint times in the UI. Can anyone explain why flink is using both?

Thanks for anyone who can help with any of the questions.

Upvotes: 2

Views: 2386

Answers (1)

David Anderson
David Anderson

Reputation: 43707

There are a lot of factors that can influence checkpointing performance, including which version of Flink you are running, which state backend you are using and how it is configured, and which kind of time windows is involved (e.g. sliding vs tumbling windows). Incremental checkpoints can have a huge impact when TBs of state are involved.

One factor that can have a large impact is the number of distinct keys involved for different time intervals. You've indicated these are keyed windows, and I would expect that over the course of an hour, many more distinct keys are used than during a typical minute. Windows are created lazily, when the first event is assigned to them, so there will be many more keyed windows created for an hour-long window than for a one-minute-long window. The same effect will arise for day-long keyed windows, but to a lesser extent.

Each of your job's operators go through a (hopefully brief) synchronous phase during checkpoint handling regardless of whether the bulk of the checkpointing is done synchronously or asynchronously. With the heap-based state backends, both synchronous and asynchronous snapshots are supported -- you'll want asynchronous snapshots for optimal performance.

Upvotes: 4

Related Questions