CODEWITHSUNDEEP
google-cloud-dataflowgoogle-cloud-mlapache-beam
Morgan Du
Morgan Du

Reputation: 61

apache_beam.runners.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed:

I set up a Google Cloud project in Cloud Shell, and tried to run this tutorial script https://github.com/GoogleCloudPlatform/cloudml-samples/blob/master/flowers/sample.sh

Ran into this error:

***@***:~/git/cloudml-samples/flowers$ ./sample.sh 
Your active configuration is: [cloudshell-4691]


Using job id:  flowers_***_20170113_162148

python trainer/preprocess.py \
  --input_dict "$DICT_FILE" \
  --input_path "gs://cloud-ml-data/img/flower_photos/eval_set.csv" \
  --output_path "${GCS_PATH}/preproc/eval" \
  --cloud
WARNING:root:Using fallback coder for typehint: Any.
WARNING:root:Using fallback coder for typehint: Any.
WARNING:root:Using fallback coder for typehint: Any.
DEPRECATION: pip install --download has been deprecated and will be removed in the future. Pip now has a download command that should be used instead.
Collecting google-cloud-dataflow==0.4.4
  Using cached google-cloud-dataflow-0.4.4.zip
  Saved /tmp/tmpSoHiTi/google-cloud-dataflow-0.4.4.zip
Successfully downloaded google-cloud-dataflow
# Takes about 30 mins to preprocess everything.  We serialize the two
Traceback (most recent call last):
  File "trainer/preprocess.py", line 436, in <module>
    main(sys.argv[1:])
  File "trainer/preprocess.py", line 432, in main
    run(arg_dict)
  File "trainer/preprocess.py", line 353, in run
    p.run()
  File "/home/slalomconsultingsf/.local/lib/python2.7/site-packages/apache_beam/pipeline.py", line 159, in run
    return self.runner.run(self)
  File "/home/slalomconsultingsf/.local/lib/python2.7/site-packages/apache_beam/runners/dataflow_runner.py", line 195, in run
    % getattr(self, 'last_error_msg', None), self.result)
apache_beam.runners.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed:
(b85b0a598a3565cb): Workflow failed.

I was not able to get any clue where I was doing wrong from the Error Log of GoogleCloud Dataflow

Appreciate any answer and help for troubleshooting.

Upvotes: 2

Views: 1459

Answers (1)

JoshGC
JoshGC

Reputation: 468

Enable the Dataflow API. In the Pantheon top search box typing "dataflow api" will take you to a window where you can click "Enable API".

I think this will fix it for you. I disabled my Dataflow API and got the same error as you, then when it was re-enabled the problem went back away.

Upvotes: 3

Related Questions