CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-dataflowgoogle-cloud-pubsub
Al Hennessey
Al Hennessey

Reputation: 2445

Cloud Pub/Sub and Cloud Dataflow fix worker to topic

I have a Google Cloud Pub/Sub and Cloud Dataflow stream processing architecture, i need guaranteed message ordering. Is it possible to set the subscription message flow so that the dataflow workers are fixed per topic, so the messages are routed through the same dataflow worker and therefore should achieve message ordering.

Thanks

Upvotes: 1

Views: 219

Answers (1)

Nathan Nasser
Nathan Nasser

Reputation: 1004

When you say you need guaranteed message ordering, do you mean: 1. Order in the final result matters in this case you can you could attach a timestamp to every event in the publisher and make the subscriber store the messages in some underlying data store. 2. Order of processed messages matters in this case you should do the same plus have a subscriber check the persistent storage to know the next message it must process.

You can read more on this in this link .

Upvotes: 0

Related Questions