CODEWITHSUNDEEP
pythonapache-kafkaconfluent-platform
antonvo
antonvo

Reputation: 213

error_cb in confluent-kafka-python producers and consumers

I'm wondering what is the proper reaction to evens that lead to the error_cb callback being called.

Initially our code was always throwing an Exception from the error_cb regardless of anything. We're running our stuff in Kubernetes, so restarting a consumer/producer is (technically) not a big deal. But the number of restart were quite significant, so we added a couple of exceptions, which we just log without quitting:

These are the ones that we see a lot, and confluent-kafka-python seems to be able to recover from them without any extra help.

Now I'm wondering if we were right to throw any exceptions in error_cb to begin with. Should we start treating error_cb just as a logging function, and only react to exceptions thrown explicitly by poll and flush?

Upvotes: 2

Views: 2843

Answers (1)

Edenhill
Edenhill

Reputation: 3113

librdkafka will do its best to automatically recover from any errors it hits, so the error_cb is mostly informational and it is generally not advisable for the application to do anything drastic upon such an error.

  • _MSG_TIMED_OUT and _TIMED_OUT- Kafka protocol requests timed out, typically due to network or broker issues. The requests will be retried according to the retry configuration, or the corresponding API / functionality willl propagate a more detailed error (e.g., failure to commit offsets). This error can safely be ignored.
  • _TRANSPORT - the broker connection went down or could not be established, again this is a temporary network or broker problem and may too be safely ignored.

Upvotes: 4

Related Questions