CODEWITHSUNDEEP
apache-kafkakafka-producer-api
shasr
shasr

Reputation: 375

Best way to configure retries in Kaka Producer

With In-Sync replicas configured as Acks=all and min.insync.replicas = N,

Want to understand how retries should be configured for the message/record for unprocessed producer records

Example: When Kafka fails to process the record with ISR online was N-1 during processing and minimum configured ISR was N replicas.

Upvotes: 0

Views: 1254

Answers (2)

Vaibs
Vaibs

Reputation: 1606

What is acks?

The acks parameter control how many partition replicas must receive the record before the producer can consider the write successful.

There are 3 values for the acks parameter:

acks=0, the producer will not wait for a reply from the broker before assuming the message sent successfully.

acks=1, the producer will receive a successful response from the broker the moment the leader replica received the message. If the message can't be written to the leader, the producer will receive an error response and can retry.

acks=all, the producer will receive a successful response from the broker once all in-sync replicas received the message.

In your case, acks=all, which is the safest way since you can make sure one more broker has the message.

Retries:

If the producer receives the error message, then the value of the retries property comes into the picture. You can use retry.backoff.ms property to configure the time between retries. It is recommended, test how long it takes to recover from a crashed broker and setting the number of retries and delay between them such that the total amount of time spent retrying will be longer than the time it takes the Kafka cluster to recover from scratch.

Also, check the below link,

https://www.confluent.io/blog/hands-free-kafka-replication-a-lesson-in-operational-simplicity/

Upvotes: 1

Rajasrikar
Rajasrikar

Reputation: 21

For above scenario, you will get NotEnoughReplicasException. At this stage, Kafka reties the message Default Retries Value as 0 for kafka <=2.0 and this value set to very high value for Kafka >=2.1 Also, there is another setting with name "retry.backof.ms". Default value for this setting is 100ms. Kafka producer will retry every 100ms to produce this message till it gets succeed. To avoid it to retry for infinite number of times or you can set "delivery.timout.ms" to make sure the Producer retries to send the message for that man millisecs. Default value for it is 120000ms. which is nothing but 2mins. Means, Producer will not retry after 2mins and considered the message as fail. Also, you might need to consider "max.in.flight.requests" setting to make sure the retry messages are processed in Sequence when your Kafka messages have a Key in it

Upvotes: 0

Related Questions