CODEWITHSUNDEEP
apache-kafkaapache-kafka-streams
Yao Dong
Yao Dong

Reputation: 143

where does kafka save the local state store?

I created a Kafka topic and sent some messages to it.
I created an application which had the stream topology. Basically, it was doing the sum and materialized it to a local state store.
I saw the new directory was created in the state folder I configured.
I could read the sum from the local state store.
Everything was good so far.
Then, I turned off my application which was running the stream.
I removed the directory created in my state folder.
I restart the Kafka cluster.
I restart my application which has the stream topology.
In my understanding, the state was gone. Kafka needs to do the aggregation again. But it did not. I was still able to get the previous sum result. How comes? Where did Kafka save the local state store?

Here is my code

Reducer<Double> reduceFunction = (subtotal, amount) -> {
    // detect when the reducer is triggered
    System.out.println("reducer is running to add subtotal with amount..." + amount);
    return subtotal + amount;
};
groupedByAccount.reduce(reduceFunction,
                Materialized.<String, Double, KeyValueStore<Bytes, byte[]>>as(BALANCE).withValueSerde(Serdes.Double()));

I explicitly put the System.out in the reduceFunction. Whenever it is executed, I shall see it on the console.
But I did not see any after restart kafka cluster and my application.
Does Kafka really recover the state? Or it saves state somewhere else?

Upvotes: 0

Views: 1297

Answers (1)

Mohammad Najar
Mohammad Najar

Reputation: 2009

If I'm not mistaken and according to Designing Event-Driven Systems by Ben Stopford (free book) on page 137, it states:

We could store, these stats in a state store and they’ll be saved locally as well as being backed up to Kafka, using what’s called a changelog topic, inheriting all of Kafka’s durability guarantees.

It seems like a copy of your state store is also backed up in Kafka itself (i.e. changelog topic). I don't think restarting a cluster will flush out (or remove) messages already in a topic as they're kept track in the Zookeeper. So once you restart your cluster and application again, the local state store is recovered from Kafka.

Upvotes: 1

Related Questions