CODEWITHSUNDEEP
google-cloud-dataflow
Graham Polley
Graham Polley

Reputation: 14791

Dataflow jobs hanging - no warnings or errors given

We've got a Dataflow pipeline that reads from BigQuery. It's trying to read ~10M records. Everything looks healthy - no errors or no warnings. But the job just hangs when trying to read from BigQuery. We left it for 15 mins, but still 0 records read. In the end we had to cancel the job.

Some example job ID's

Is there something wrong with the service currently?

Upvotes: 1

Views: 276

Answers (1)

Graham Polley
Graham Polley

Reputation: 14791

It was a permission problem reading from a BigQuery table. However, the errors were not visible in the job. When we enabled cloud logging, then the job failed and the errors showed up. The Google engineers are looking into why the job hangs when cloud logging is not enabled, and when an error occurs.

Upvotes: 1

Related Questions