CODEWITHSUNDEEP
apache-flinkflink-streamingamazon-kinesis-analytics
Swapnil Khante
Swapnil Khante

Reputation: 659

Apache Flink - Checkpointed Data Size is increasing over the period of time

I am using Event time semantics in my Flink application (version 1.11.1) which is running in AWS - kinesis analytics. This application has source as kinesis stream and sink as Postgres. Checkpointing interval is 10 seconds as DB sink is triggered on notifyCheckpointComplete(). I am using multiple CoProcessFunction along with ValueState to connect different streams before I sink it to Postgres.

Observation is the Checkpointed Data Size is growing over the period of time whereas thread count and Heap memory utilization remains constant. CPU utilization does not go beyond 30 percent. I am hoping the checkpoint data size to plateau eventually.

enter image description here

While going through flink documentation on State TTL, it seems that currently state ttl only supported for Processing time semantics - State Time-To-Live (TTL)

What is the way forward for Event time based Flink application?

Upvotes: 0

Views: 996

Answers (1)

David Anderson
David Anderson

Reputation: 43697

You can still use State TTL. It's just that the state retention interval has to be expressed in terms of wall-clock time, rather than being related to the timestamps in the events.

But if you want more control over when and how the state is cleared, you can use timers in your CoProcessFunctions to trigger explicit calls to clear.

Upvotes: 0

Related Questions