CODEWITHSUNDEEP
google-cloud-dataflow
Echo
Echo

Reputation: 405

Google dataflow failure

Been testing dataflow for a little while, today I run into a few failures, error messages were:

Causes: (b8a42e32d0888f60): Unable to rename ouput files from gs://clickstream_current/tmp/dataflow/staging/clickstream/8977742977819433140/dax-tmp-2015-04-14_15_58_06-5441905562239213266-S01-1-e70068cb69ef966a/@DAX.json to gs://clickstream_current/tmp/dataflow/staging/clickstream/8977742977819433140/@*.json. Causes: (b8a42e32d0888fdb): Unable to rename "gs://clickstream_current/tmp/dataflow/staging/clickstream/8977742977819433140/dax-tmp-2015-04-14_15_58_06-5441905562239213266-S01-1-e70068cb69ef966a/-shard-00000-of-00940-endshard.json" to "gs://clickstream_current/tmp/dataflow/staging/clickstream/8977742977819433140/-00000-of-00940.json.

Could this be GCS issue? besides the failure there were warning complaining about "Unable delete temporary files from GCS folders" anything I can do to avoid it?

Upvotes: 0

Views: 464

Answers (1)

Jeremy Lewi
Jeremy Lewi

Reputation: 6776

We've identified an issue in the service that will cause this failure in certain rare circumstances. We are working on fixing this issue. In the meantime we apologize for the inconvenience. The error is slightly more likely if you are using a BigQueryIO.Write to output your data.

Upvotes: 1

Related Questions