CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-dataflowgoogle-cloud-pubsub
user3483129
user3483129

Reputation: 137

Stop Streaming pipeline when no more messages to consume

I have a streaming dataflow pipeline job which reads messages from a given pub-sub topic. I understand there is an auto-ack once the bundles are committed. How to make the pipeline stop where there are no more messages to consume?

Edit - I have a scenario where I need to drain off incorrect messages sent to the topic. Thus this would a one time job. My application sends 1MM messages only once a day (fixed time) to that topic.

Upvotes: 1

Views: 583

Answers (1)

Christopher
Christopher

Reputation: 941

Why would you want to stop the streaming pipeline, concerned about the being charged when the pipeline is doing nothing? If this is your concern then you should not be concerned at all since you will only be charged for the resources you use e.g CPU hour, Memory, Disk Storage, etc. please see pricing details here

Since your source is unbounded (e.g PubSub) then there's no way you could tell that there will be no more incoming data in the future.

Upvotes: 1

Related Questions