CODEWITHSUNDEEP
apache-kafkakafka-producer-api
Nag
Nag

Reputation: 2047

Kafka - Message Ordering Guarantees

I come across two phrases with respect to ordering,

  1. Messages sent by a producer to a particular topic partition will be appended in the order they are sent. That is, if a record M1 is sent by the same producer as a record M2, and M1 is sent first, then M1 will have a lower offset than M2 and appear earlier in the log.

Another

  1. (config param) max.in.flight.requests.per.connection - The maximum number of unacknowledged requests the client will send on a single connection before blocking. Note that if this setting is set to be greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (i.e., if retries are enabled).

The question is, will the order still be retained to a particular partition if there are failed sends like mentioned #2 ? if there is a potential issue with one message , all the following messages will be dropped "to retain the order" per partition or the "correct" messages will be sent and failed messages will be notified to the application ?

Upvotes: 2

Views: 1916

Answers (1)

Michael Heil
Michael Heil

Reputation: 18475

"will the order still be retained to a particular partition if there are failed sends like mentioned #2?"

As written in the documentation part you have copied, there is a risk that the ordering is changed.

Imagine, you have a topic with e.g. one partition. You set the retries to 100 and the max.in.flight.requests.per.connection to 5 which is greater than one. As a note, retries will only make sense if you set the acks to 1 or "all".

If you plan to produce the following messages in the order K1, K2, K3, K4, K5 and it takes your producer some time to

  • actually create the batch and
  • make a request to the broker and
  • wait for the acknowledgement of the broker

you could have up to 5 requests in parallel (based on the setting of max.in.flight.request.per.connection). Now, producing "K3" has some issues and it goes into the retry-loop, the messages K4 and K5 can be produced as the request was already in flight.

Your topic would end up with messages in that order: K1, K2, K4, K5, K3.

In case you enable idempotency in the Kafka Producer, the ordering would still be guaranteed as explained in Ordering guarantees when using idempotent Kafka Producer

Upvotes: 4

Related Questions