CODEWITHSUNDEEP
javaapache-kafkaapache-kafka-streams
Oskar Polak
Oskar Polak

Reputation: 161

Is Kafka streams aggregation function stores data both in ram and topic?

I'm new in kstreams, and want to understand some of processes.

Aggregation function is creating a kafka topic with changelog, but why? Only for backup? As far as I understand kstreams needs statestore locally, so when is used data from topic? Only if some operations fault or to recreate statestore after app restart?

Upvotes: 0

Views: 90

Answers (1)

Bartosz Wardziński
Bartosz Wardziński

Reputation: 6593

Changelog topics are created for state store. Although state store is kept locally, its value might be used remotely.

If you performed aggregation several situation can happen:

  1. Application can crashed.
  2. You can stop you application
  3. Rebalance might happen.
  4. etc

In those situation intermediate results are needed to calculate final one. If there wasn't changelog topic with intermediate results, computation would have to start from scratch.

I think changelog topic is for performance and fault tolerance.

Some interesting information about Internal topic and Duality of streams and tables(changelog topic).

Upvotes: 2

Related Questions