CODEWITHSUNDEEP
apache-kafkakafka-producer-apikafka-python
user3851499
user3851499

Reputation:

Does Kafka guarantee message ordering within a single partition with ANY config param values?

If I set Kafka config param at Producer as:

1. retries = 3
2. max.in.flight.requests.per.connection = 5

then its likely that Messages within one partition may not be in send_order.

Does Kafka takes any extra step to make sure that messages within a partition remains in sent order only OR With above configuration, its possible to have out of order messages within a partition ?

Upvotes: 9

Views: 1945

Answers (1)

serejja
serejja

Reputation: 23841

Unfortunately, no.

With your current configuration, there is a chance message will arrive unordered because of your retries and max.in.flight.requests.per.connection settings..

With retries config set to greater than 0 you will lose ordering in the following case (just an example with random numbers):

  1. You send a message/batch to partition 0 which is located on broker 0, and brokers 1 and 2 are ISR.
  2. Broker 0 fails, broker 1 becomes leader.
  3. Your message/batch returns a failure and needs to be retried.
  4. Meanwhile, you send next message/batch to partition 0 which is now known to be on broker 1, and this happens before your previous batch actually gets retried.
  5. Message/batch 2 gets acknowledged (succeeds).
  6. Message/batch 1 is re sent and now gets acknowledged too.
  7. Order lost.

I might be wrong, but in this case, reordering can probably happen even with max.in.flight.requests.per.connection set to 1 you can lose message order in case of broker failover, e.g. batch can be sent to the broker earlier than the previous failed batch figures out it should go to that broker too.

Regarding max.in.flight.requests.per.connection and retries being set together it's even simpler - if you have multiple unacknowledged requests to a broker, the first one to fail will arrive unordered.

However, please take into account this is only relevant to situations where a message/batch fails to acknowledge for some reason (sent to wrong broker, broker died etc.)

Hope this helps

Upvotes: 14

Related Questions