CODEWITHSUNDEEP
javaapache-kafkakafka-consumer-apiapache-kafka-streamsspring-kafka
Norbert Koch
Norbert Koch

Reputation: 541

After kafka crashed, the offsets are lost

Our kafka system crashed because no disk space was available. The consumers are Spring boot application which are using the Kafka Streams API. Now every consumer application shows the following error:

java.io.FileNotFoundException: /tmp/kafka-streams/908a79bc-92e7-4f9c-a63a-5030cf4d3555/streams.device-identification-parser/0_48/.checkpoint.tmp (No such file or directory)

This exception occurred exactly after the kafka server was restarted. If we restart the application, the service starts at offset 0 to replay all message from the consumed topic. This is fatal.

We checked the logs: On each partition there is no information (example line, we have 60 partitions):

TOPIC                               PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                                                                                            HOST            CLIENT-ID
device-identification.received      0          -               0               -               streams.device-identification-parser-3e81ef05-b9e3-495c-85b3-7e2912f00578-StreamThread-1-consumer-ca723291-3258-46c0-8b23-233bdc1e1191 /XX.X.X.X       streams.device-identification-parser-3e81ef05-b9e3-495c-85b3-7e2912f00578-StreamThread-1-consumer

Can anyone help me, to restore the offset information, that the service starts from the last offset?

Upvotes: 3

Views: 1601

Answers (1)

Chris Matta
Chris Matta

Reputation: 3443

If this is a Kafka Streams app you might be successful in resetting it, that can be done using Confluent's Application reset tool: https://docs.confluent.io/current/streams/developer-guide/app-reset-tool.html

Upvotes: 1

Related Questions