CODEWITHSUNDEEP
google-bigquerygoogle-cloud-dataflowgoogle-cloud-pubsub
Jonny5
Jonny5

Reputation: 1499

Duplicates in Pub/Sub-Dataflow-BigQuery pipeline

Consider the following setup:

We have counters on the valid events that pass through our Datafow Pipeline and observe the counters are higher than the amount of events that were available in Pub/Sub.

Note: It seems we also see duplicates in BigQuery, but we are still investigating this.

The following error can be observed in the Dataflow logs:

Pipeline stage consuming pubsub took 1h35m7.83313078s and default ack deadline is 5m. 
Consider increasing ack deadline for subscription projects/<redacted>/subscriptions/<redacted >

Note that the Dataflow job is started when there are already millions of messages waiting in Pub/Sub.

Questions:

Upvotes: 1

Views: 549

Answers (1)

guillaume blaquiere
guillaume blaquiere

Reputation: 75715

My recommendation is to purge the PubSub subscription message queue by launching the Dataflow job in batch mode. Then run it in streaming mode for the usual operation. Like this, you will be able to start from a clean basis your streaming job and not have a long list of enqueued messages.

In addition, it's the power of Dataflow (and beam) to be able to run in streaming and in batch the same pipeline.

Upvotes: 1

Related Questions