CODEWITHSUNDEEP
apache-kafkaapache-flink
Skala
Skala

Reputation: 1

Processing and alerting on no(zero) events in a flink process window

We are building a flink use case where we are consuming from a kafka topic and performing aggregations and generating alerts based on average, max, min thresholds. We also need to notify for each key when there are 0 events in a Tumbling Event Time Windows. We are having trouble coming up with a solution to do the same. The options we considered are below, please let us know if there are other ideas we haven't looked into.

  1. Queryable State : Save the keys in each of the Process Window Functions. Query the state from an external application and alert when a key is missing after the 20min time interval has expired. We see Queryable state feature is being deprecated in the future. We do not want to go down this path when we already know there is an EOL for it.
  2. Use Processing Time Windows : Using Processing time instead of Event time would have been an option if our downstream applications would send out events in real time. Maintenances of the downstream applications, delays etc would result in a lot of data loss which is undesirable.
  3. Generate artificial kafka events: We unfortunately do not have permissions to generate artificial events in our ecosystem.

Flink version : 1.14.3

Due the number of events being processed per window, we are using incremental aggregate function https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/windows/#processwindowfunction-with-incremental-aggregation

Thanks,

Sweta

Upvotes: 0

Views: 265

Answers (1)

David Anderson
David Anderson

Reputation: 43657

This can be solved by introducing a KeyedProcessFunction into the pipeline that uses timers to create canary events that will cause the otherwise empty windows to exist. These events can then be filtered out of the results.

See this thread on the user mailing list for more.

Upvotes: 0

Related Questions