CODEWITHSUNDEEP
apache-kafka-streams
Viswapriya
Viswapriya

Reputation: 191

KafkaStreams shuts down with no exceptions

I have four instances of a Kafka stream application running with the same application id. All the input topics are of a single partition. To achieve scalability I have passed it through an intermediate dummy topic with multiple partitions. I have set request.timeout.ms as 4 minutes.

The Kafka instances go into the ERROR state without any exception being thrown. It is difficult to figure out what is the exact issue. Any ideas?

[INFO ] 2018-01-09 12:30:11.579 [app-new-03-cb952917-bd06-4932-8c7e-62986126a5b4-StreamThread-1] StreamThread:939 - stream-thread [app-new-03-cb952917-bd06-4932-8c7e-62986126a5b4-StreamThread-1] Shutting down
[INFO ] 2018-01-09 12:30:11.579 [app-new-03-cb952917-bd06-4932-8c7e-62986126a5b4-StreamThread-1] StreamThread:888 - stream-thread [app-new-03-cb952917-bd06-4932-8c7e-62986126a5b4-StreamThread-1] State transition from RUNNING to PENDING_SHUTDOWN.
[INFO ] 2018-01-09 12:30:11.595 [app-new-03-cb952917-bd06-4932-8c7e-62986126a5b4-StreamThread-1] KafkaProducer:972 - Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
[INFO ] 2018-01-09 12:30:11.605 [app-new-03-cb952917-bd06-4932-8c7e-62986126a5b4-StreamThread-1] StreamThread:972 - stream-thread [app-new-03-cb952917-bd06-4932-8c7e-62986126a5b4-StreamThread-1] Stream thread shutdown complete
[INFO ] 2018-01-09 12:30:11.605 [app-new-03-cb952917-bd06-4932-8c7e-62986126a5b4-StreamThread-1] StreamThread:888 - stream-thread [app-new-03-cb952917-bd06-4932-8c7e-62986126a5b4-StreamThread-1] State transition from PENDING_SHUTDOWN to DEAD.
[WARN ] 2018-01-09 12:30:11.605 [app-new-03-cb952917-bd06-4932-8c7e-62986126a5b4-StreamThread-1] KafkaStreams:343 - stream-client [app-new-03-cb952917-bd06-4932-8c7e-62986126a5b4] All stream threads have died. The Kafka Streams instance will be in an error state and should be closed.
[INFO ] 2018-01-09 12:30:11.605 [new-03-cb952917-bd06-4932-8c7e-62986126a5b4-StreamThread-1] KafkaStreams:268 - stream-client [app-new-03-cb952917-bd06-4932-8c7e-62986126a5b4] State transition from RUNNING to ERROR.

Upvotes: 16

Views: 10036

Answers (2)

Mikael Carlstedt
Mikael Carlstedt

Reputation: 24

You may also need to set the default.production.exception.handler Kafka Streams property to a class that implements ProductionExceptionHandler and, unlike the default class DefaultProductionExceptionHandler, logs the error before triggering a permanent failure state.

Upvotes: -1

Dennis Jaheruddin
Dennis Jaheruddin

Reputation: 21561

Thea asker shared his solution in the comments:

Once I changed the consumer group id it worked.

It is also worth noting that the related issue (which may or may not have the same root cause) has been introduced in recent versions, and now looks to be fixed as well in Kafka versions 2.5.1 and 2.6.0 above.

As such people who are experiencing this today may want to check whether they are on a high (or low) enough version to avoid this issue.

Upvotes: 3

Related Questions