CODEWITHSUNDEEP
apache-kafkakafka-consumer-apikafka-producer-api
nilesh1212
nilesh1212

Reputation: 1655

How is message sequence preserved for topic with many partitions?

I want any information/explanation on how Kafka maintains a message sequence when messages are written to topic with multiple partition. For e.g. I have multiple message producer each producing messages sequentially and writing on the Kafka topic with more than 1 partition. In this case, how consumer group will work to consume messages.

Upvotes: 3

Views: 3473

Answers (2)

amethystic
amethystic

Reputation: 7091

Kafka only provides a total order over records within a partition, not between different partitions in a topic. Even within one partition, you still could encounter the out-of-order events if retries is enabled and max.in.flight.requests.per.connection is larger than 1.

Work-around is create a topic with only one partition although it means only one consumer process per consumer group.

Upvotes: 6

Kraylog
Kraylog

Reputation: 7563

Kafka will store messages in the partitions according to the message key given to the producer. If none is given, then the messages will be written in a round-robin style into the partitions. To keep ordering for a topic, you need to make sure the ordered sequence has the same key, or that the topic has only one partition.

Upvotes: 5

Related Questions