CODEWITHSUNDEEP
javaapache-kafkamessage-queueapache-kafka-streamsdistributed-computing
yongjieyongjie
yongjieyongjie

Reputation: 893

Waiting for N Events in Kafka: How does the Group-Count-Filter Approach work?

Background

In the book Designing Event-Driven Systems, the author highlighted a common use case in business systems where there is a need to wait or N events to occur.

The example given is of an orders service that needs to wait for three separate validation services, all sent via the same topic, to return a PASS. (I interpret this to mean that there will be three validation messages on the same topic with the same key, each with a value indicating success or failure.)

The author stated that the solution will take the following form (assuming that the counting is based on the key):

  1. Group by the key.
  2. Count occurrences of each key (using an aggregator executed with a window).
  3. Filter the output for the required count.

Question

How exactly does each of the step above works, and what are the classes / methods involved?

Upvotes: 1

Views: 1233

Answers (1)

Matthias J. Sax
Matthias J. Sax

Reputation: 62350

I guess your assumptions are right. For step (2) it depends, but if you assume that all the messages you are waiting for have the same key and you are only interested if three messages with the same key got received, calling count() is what you want.

As a next step (ie, (3)), you would call KTable#filter() to get all rows with a count of 3.

Finally, you can call toStream() and this stream should contain a record each time one key reaches the count of 3.

(As a side remarks: all entries would stay in the KTable forever by default, thus you also need to take care to delete entries that did reach count of 3 at some point.)

Upvotes: 0

Related Questions