CODEWITHSUNDEEP
apache-kafkaspring-kafkaspring-cloud-stream
Jacob Wallace
Jacob Wallace

Reputation: 927

How can a Kafka consumer maintain strict ordering when errors occur?

Spring for Apache Kafka supports non-blocking retries using retry topics, with the important caveat that ordering guarantees are lost for retried messages.

Confluent describes a pattern for maintaining the order of retried events by augmenting a retry topic with the new concept of a redirect topic.

Are there any reference implementations of this pattern, either open source (e.g., using spring-kafka) or on cloud vendors (e.g., Confluent, MSK)? Thanks!

Upvotes: 0

Views: 1142

Answers (1)

Gary Russell
Gary Russell

Reputation: 174729

You cannot use non-blocking retries and retain strict ordering of records within a partition.

If you mean you want to maintain strict ordering within a subset of the partition contents (e.g. records with the same key). It is simpler to NOT use non-blocking retries and significantly increase the number of partitions so that other records within other partitions can be processed while reading from the failed partition is blocked.

Trying to retain strict ordering of a partition subset would be non-trivial.

Upvotes: 2

Related Questions