CODEWITHSUNDEEP
pythonapache-kafkaconfluent-platform
FBryant87
FBryant87

Reputation: 4595

Prevent Confluent Kafka from losing messages when producing

The Confluent Kafka library (python version in this case) has a produce method which takes a delivery callback function:

kafka_producer.produce(topic=topic,
                            key=key,
                            value=value,
                            on_delivery=delivery_callback)

This callback is called whether the message was successfully delivered or not:

def delivery_callback(err, msg):

I don't have any retry logic in this function if the message failed, because the docs say it is asynchronous.

Instead every 100 messages or so, I rely on flush() to tell me if any messages weren't successfully produced:

messages_outstanding = kafka_producer.flush()
if messages_outstanding == 0:
   //continue to the next batch of 100
else:
   //produce the batch again

Will flush() account for any messages which failed to produce? (reported as errors in delivery_callback)

In other words, can I be sure flush() won't return zero if any messages failed?

Upvotes: 4

Views: 1846

Answers (1)

FBryant87
FBryant87

Reputation: 4595

Confirmed the following results:

Calling .flush() can definitely return zero even if messages failed to produce. This method appears to wait until all delivery callbacks have finished for all messages (a callback can simply report that the message failed to deliver).

The whole thing's surprisingly awkward from our perspective. If you can't afford to lose messages, you need to detect when a delivery callback fails, and implement some form of retry logic to cover the failed messages.

Upvotes: 1

Related Questions