CODEWITHSUNDEEP
streamingapache-flinkflink-streamingpyflink
whatsinthename
whatsinthename

Reputation: 2157

Flink developer's role from watermarks perspective

I am a newbie to Flink and came across an article that mentioned

"A flink developer is responsible for moving event time forward by arranging the watermark in the stream".

So, I figured out the possible answer for this. As per my knowledge, if I Instruct the program to emit watermarks every 5 seconds. Actually, every 5 seconds, Flink invokes the getCurrentWatermark() method of AssignerWithPeriodicWatermarks. If the method returns a non-null value with a timestamp larger than the timestamp of the previous watermark, the new watermark is forwarded. This check is necessary to ensure event time continuously increases; otherwise, no watermark is produced.

So, once everything within a window has arrived it will trigger the operators and computations will be done accordingly and what is the role of the processfunctions? Watermarks can be used by processfunctions only, right?

Upvotes: 1

Views: 465

Answers (1)

David Anderson
David Anderson

Reputation: 43474

What you've said about periodic watermarks is correct. But normally I would recommend leaving the autowatermarking interval at its default value of 200 msec; setting it to 5 seconds will add 5 seconds of latency to your pipeline.

At the lowest levels of Flink's APIs, watermarks serve to trigger event-time timers, which are only exposed in process functions. Process functions are an essential building block for implementing event-driven applications. You are hooking right into the main event loop, processing each event as it becomes available. You also have access to fault-tolerant, low-latency, scalable state storage, and timers.

At higher levels of the DataStream API, watermarks are used to trigger event-time windows, and by CEP (to sort streams before doing pattern matching). Watermarks are also used in the Table/SQL API by windows, interval joins, temporal joins, and by MATCH_RECOGNIZE. In all of these cases, watermarks are used by these temporal operators to observe the progress of event time so they can emit results when those results are ready, and then to free state that is no longer useful.

Upvotes: 1

Related Questions