CODEWITHSUNDEEP
google-cloud-dataflowgoogle-cloud-pubsub
Brachi
Brachi

Reputation: 757

Drain DataFlow job and start another one right after, cause to message duplication

I have a dataflow job, that subscribed to messages from PubSub:

p.apply("pubsub-topic-read", PubsubIO.readMessagesWithAttributes()


.fromSubscription(options.getPubSubSubscriptionName()).withIdAttribute("uuid"))

I see in docs that there is no guarantee for no duplication, and Beam suggests to use withIdAttribute.

This works perfectly until I drain an existing job, wait for it to be finished and restart another one, then I see millions of duplicate BigQuery records, (my job writes PubSub messages to BigQuery).

Any idea what I'm doing wrong?

Upvotes: 1

Views: 655

Answers (1)

chamikara
chamikara

Reputation: 2024

I think you should be using the update feature instead of using drain to stop the pipeline and starting a new pipeline. In the latter approach state is not shared between the two pipelines, so Dataflow is not able to identify messages already delivered from PubSub. With update feature you should be able to continue your pipeline without duplicate messages.

Upvotes: 3

Related Questions