CODEWITHSUNDEEP
apache-kafkakafka-producer-api
Nag
Nag

Reputation: 2057

Kafka ordering guarantees

I was going through kafka documentation and came across

Guarantees

At a high-level, Kafka gives the following guarantees:

Messages sent by a producer to a particular topic partition will be appended in the order they are sent. That is, if a record M1 is sent by the same producer as a record M2, and M1 is sent first, then M1 will have a lower offset than M2 and appear earlier in the log. A consumer instance sees records in the order they are stored in the log. For a topic with replication factor N, we will tolerate up to N-1 server failures without losing any records committed to the log.

I had few questions.

  1. Is it always guaranteed that M1 will have a lower offset than M2 ? what if M1 is retried later than M2 ?
  2. I also understood from various documentations that ordering is not guaranteed, and the consumer has to deal with it.

Upvotes: 4

Views: 12113

Answers (3)

c.guzel
c.guzel

Reputation: 301

I have an article about deep understanding ordering guarantees provided by Kafka. You can check it in my medium post.

Upvotes: -3

Mariusz
Mariusz

Reputation: 13926

Please notice, that ordering guarantees apply at the partition level. So, if you have more than one partition in the topic, you'll need to set the same partition key for messages that you require to appear in order.

For example, if you want to collect messages from various sensors and sensor has it's id, then if you use this ID as message key, ordering of messages from every sensor will be guaranteed on consumers (as no sensor will write messages to more than 1 partition).

To answer your questions:

  1. Yes, M1 will have always offset lower than M2. The offsets are set by broker, so the time of message arrival at the broker is key here.
  2. Ordering is not guaranteed on topic level only

Upvotes: 7

vahid
vahid

Reputation: 1218

A possible scenario even with a single partition is:

  • Producer sends M1
  • Producer sends M2
  • M1 is not ack'ed on the first try due to some failure
  • M2 is delivered
  • M1 is delivered in a subsequent try.

One easy way to avoid this is through the producer config max.in.flight.requests.per.connection=1.

This of course has performance implications, so it should be used with caution.

Upvotes: 8

Related Questions