CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-dataflow
SarangRN
SarangRN

Reputation: 140

Once dataflow job is created in Google Cloud using apache beam sdk, can we delete the tmp files from cloud storage bucket?

Dataflow job has been created to store the logs details from google dialogflow to big query enter image description here

Upvotes: 2

Views: 818

Answers (1)

Sayan Bhattacharya
Sayan Bhattacharya

Reputation: 1368

Dataflow jobs do not depend on any temporary files once the job has been stopped. However, deleting the temporary files of a running job would cause problems.

Therefore, I recommend the following. If you do not have any dataflow job running that are using a particular GCS location for storing temporary files,you can delete all the temp files in that GCS location to avoid costs.
If you have any dataflow jobs that are still running, do not delete any of the temporary files in their GCS temp files location. If you do not wish to manually delete files, you can use an object lifecycle policy to move those temporary files once the dataflow stops.

Upvotes: 2

Related Questions