CODEWITHSUNDEEP
apache-kafkaapache-kafka-streams
tuna
tuna

Reputation: 312

Kafka Streams state restoration on Exception

With kafka streams we can do stateful operations. The state is kept local to the instance. This state is also backed up in a changelog topic, in case you lose your local state.

I saw that full state restoration also happens in case of an unhandeled exception in the streams application. Even if you use persistent storage. Imagine if you have terrabytes of state and due a nullpointer exception, kafka will begin with the restoration of this huge amount of data. This will be very time consuming.

Why does kafka do a full restore of the state after an unhandled exception? Can we avoid this? Wrapping the code in a try-catch is not an option, because the message is then considered succesfully processed. We do some complex logic and also save some values in the state store.

Upvotes: 1

Views: 921

Answers (1)

groo
groo

Reputation: 4438

By design the default is that runtime exceptions will stop your streaming process but you can control that to some extent using Exception Handlers -> https://developer.confluent.io/learn-kafka/kafka-streams/error-handling/

So you can for example send that specific message to a dead letter queue to process or check later what is wrong.

You can also configure the state store replication to avoid long / costly restoration times using num.standby.replicas, check -> https://kafka.apache.org/30/documentation/streams/architecture#streams_architecture_recovery

Good luck!

Upvotes: 2

Related Questions