CODEWITHSUNDEEP
javaapache-kafkaapache-kafka-streams
Victor Denisov
Victor Denisov

Reputation: 747

Emitting values from ktable and modifying it

I'm trying to solve the following problem with kafka. There is a topic. let's call it src-topic. I receive records from this topic from time to time. I would like to store those values in a ktable and emit the values stored in the ktable every 10 seconds to dst-topic. When I emit a value from this ktable for the first time then I want to append 1 to the record I emit. Every subsequent time I would like to append 0 to the emitted record.

I'm looking for a correct and preferably idiomatic solution to this issue. One of the solutions I see is to emit a record with 1 appended when I ingest from src-topic and then store in the ktable the record with 0 appended. Another thread will be reading from this ktable and emitting the records regularly. The problem with this approach is that it has a race condition.

Any advice will be appreciated.

Upvotes: 0

Views: 556

Answers (1)

Matthias J. Sax
Matthias J. Sax

Reputation: 62330

There is no straight forward way to do this. Note, a KTable is a changelog stream (it might have a table state internally -- not all KTables do have a state --, but that's an implementation detail).

Thus, a KTable is a stream and you cannot flush a stream... And because the state (if there is any) is internal, you cannot flush the state either.

You can only access the state via Interactive Queries that also allow to do a range scan. However, this will not emit anything downstream but gives the data to the "non Streams part" of you application.

I think, you will need to use low-level Processor API to get the result you want.

Upvotes: 1

Related Questions