CODEWITHSUNDEEP
spring-bootapache-kafkakafka-producer-api
JustCode
JustCode

Reputation: 61

How Kafka retries works with request.timeout.?

I have configured my Producer with request.timeout.ms = 70,0000ms and retries=5. I have doubt how this actually works,

After this "request.timeout.ms=70,000" expires it retries for 5 times or within given "request.timeout.ms=70,000" it retries for 5 time with retry.backoff.ms value.?

Upvotes: 3

Views: 7366

Answers (1)

Gwen Shapira
Gwen Shapira

Reputation: 5168

There are 3 important configs to be aware of:

  • "request.timeout.ms" - time to retry a single request
  • "delivery.timeout.ms" - time to complete the entire send operation
  • "retries" - how many times to retry when the broker responds with retriable errors.

The Apache Kafka recommendation is to set "delivery.timeout.ms" and leave the other two configurations with their default value. The idea is that the main thing you as a user should worry about is how long you want to way for Kafka to figure things out before giving up on it. It doesn't really matter what is taking Kafka so long - the connection, getting metadata, long queues, etc, the only thing that matters is how long you are willing to wait.

Now to your question - request.timeout.ms applies on each retry. So Producer will send the recordbatch to Kafka, and if there's no response after 70,000ms it will consider this a failure and retry. Note that most errors (say, NoLeaderForPartition) will return from the broker much faster (which is why retry backoffs are needed).

Reasoning about delivery times with retries + request.timeout.ms turned out to be near impossible even for those who wrote the producer. Hence, the introduction of delivery.time.ms with a very clear contract.

Upvotes: 7

Related Questions