CODEWITHSUNDEEP
apache-kafkaapache-kafka-streams
arunmahadevan
arunmahadevan

Reputation: 368

How does kafka streams compute watermarks?

Does Kafka Streams internally compute watermarks? Is it possible to observe the result of a window (only) when it completes (i.e. when watermark passes end of window)?

Upvotes: 6

Views: 2555

Answers (2)

Matthias J. Sax
Matthias J. Sax

Reputation: 62330

Does Kafka Streams internally compute watermarks?

No. Kafka Streams follows a continuous update processing model that does not require watermarks. You can find more details about this online:

Is it possible to observe the result of a window (only) when it completes (i.e. when watermark passes end of window)?

You can observe of result of a window at any point in time. Either via subscribing to the result changelog stream via for example KTable#toStream()#foreach() (ie, a push based approach), or via Interactive Queries that let you query the result window actively (ie, a pull based approach).

As mentioned by @Dmitry, for the push based approach, you can also use suppress() operator if you are only interested in a window's final result.

Upvotes: 3

Dmitry Minkovsky
Dmitry Minkovsky

Reputation: 38143

Kafka Streams does not use watermarks internally, but a new feature in 2.1.0 lets you observe the result of a window when it closed. It's called Suppressed, and you can read about it in the docs: Window Final Results:

KGroupedStream<UserId, Event> grouped = ...;
grouped
    .windowedBy(TimeWindows.of(Duration.ofHours(1)).grace(ofMinutes(10)))
    .count()
    .suppress(Suppressed.untilWindowCloses(unbounded()))

Upvotes: 4

Related Questions