CODEWITHSUNDEEP
scalaapache-kafkaapache-kafka-streams
Mon Calamari
Mon Calamari

Reputation: 4463

Evolution of kafka-streams topology in production

I have a complex topology including KStreams, KTables, joins, rekeying, filters, through, branching, zips etc with live clients on output topics.

All the input, intermidate and output topics are avro based topics.

How do I handle changes to the topology (adding new steps, new input-intermidate-output topics), knowing kafka-streams maintains internal state and changelogs, complying with the following business requirements:

Is the answer lying somewhere between using kafka-streams-application-reset or creating brand new consumer group (application id)?

Upvotes: 3

Views: 1012

Answers (1)

Nishu Tayal
Nishu Tayal

Reputation: 20820

At the moment, there is no checkpointing or savepoint type concept in Kafka Streams.

If you need to update the existing applications, it is safe to use the new application.id or reset the existing application.id.

There are scenarios where you need to define a new application.id based on the changes in topology.

https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams+Data+%28Re%29Processing+Scenarios

Upvotes: 5

Related Questions