CODEWITHSUNDEEP
springcloud-foundryspring-cloud-streamspring-cloud-dataflow
user6448130
user6448130

Reputation:

Spring Cloud Dataflow - Retaining Order of Messages

Let's say I have a stream with 3 applications - a source, processor, and sink.

I need to retain the order of my the messages I received from my source. When I receive messages A,B,C,D, I have to send them to sink as A,B,C,D. (I can't send them as B,A,C,D).

If I have just have 1 instance of each application, everything will run sequentially and the order will be retained.

If I have 10 instances of each application, the messages A,B,C,D might get processed at the same time in different instances. I don't know what order these messages will wind up in.

So is there any way I can ensure that I retain the order of my messages when using multiple instances?

Upvotes: 1

Views: 324

Answers (1)

Gary Russell
Gary Russell

Reputation: 174584

No; when you scale out (either by concurrency in the binder or by deploying multiple instances), you lose order. This is true for any multi-threaded application, not just spring-cloud-stream.

You can use partitioning so that each instance gets a partition of the data, but ordering is only retained within each partition.

If you have sequence information in your messages, you can add a custom module using a Spring Integration Resequencer to reassemble your messages back into the same sequence - but you'll need a single instance of the resequencer before a single sink instance.

Upvotes: 3

Related Questions