CODEWITHSUNDEEP
javagoogle-cloud-dataflowapache-beam
Michał
Michał

Reputation: 636

What is the easiest way to save file from Google Dataflow to Google Storage?

I've got my pipeline up and running, but I need to save a number to a file, which will allow me to come back to previous processing in case something goes wrong ( or when I want to deploy new version of pipeline, and I cannot use the update-pipeline feature ).

I was trying to add libraries from google-cloud-storage, but there are errors in Runtime (it does not find some classes from google-cloud-core even when I have all dependencies up to date ). There need to be a simple way, just to save ( and then read ) a file. I will be grateful for some snippet or link.

Upvotes: 0

Views: 258

Answers (2)

Saman
Saman

Reputation: 151

Probably it is too late to answer this :D. But perhaps can help other people. If you are encountering dependency hell because of using different GCP components, use https://github.com/googleapis/google-cloud-java/tree/master/google-cloud-bom. They have created a pom file that keeps track of all the compatible versions in different components.

Upvotes: 1

CasualT
CasualT

Reputation: 5049

If you are stuck in dependency hell, and must do this step from within dataflow, then I would recommend looking at side-inputs and outputs...and then pairing that with some simple TextIO operations.

Upvotes: 0

Related Questions