CODEWITHSUNDEEP
apache-kafkakafka-consumer-api
dunn less
dunn less

Reputation: 623

Kafka partitions order of consumption

Lets assume that i have a single consumer and a topic with 2 partitions A and B

If the consumer is connected to both of these partitions, in which order will the messages be consumed?

e.g

Partition A

-------------
| A | B | C |
-------------

Partition B

-------------
| X | Y | Z |
-------------

Will the messages be consumed randomly e.g 

A -> X -> Y -> B

Or is there any mechanisms (round-robin) that is applied 

A -> X -> B -> Y -> C -> Z

Upvotes: 1

Views: 62

Answers (1)

Giorgos Myrianthous
Giorgos Myrianthous

Reputation: 39930

Kafka guarantees the order of messages only within a single partition.

The simplest way (not the most efficient one, though) to deal with this, is to create a single partition and run a single consumer.

Alternatively, you can partition the messages based on their key. Messages with the same key are inserted to the same partition and therefore the order of messages with the same key is guaranteed. For example, let's say that your data stream contains users' details. You can use e.g. userID as key, so that all of the messages for a user with a particular userID are inserted to the same partition. The consumer will then consume the messages of the user with key=userID in the exact order they have been inserted to the topic (or the partition to be more precise).

Upvotes: 2

Related Questions