CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-dataflow
manoj
manoj

Reputation: 11

Data flow pipeline got stuck

Workflow failed. Causes: The Dataflow job appears to be stuck because no worker activity has been seen in the last 1h. Please check the worker logs in Stackdriver Logging. You can also get help with Cloud Dataflow at https://cloud.google.com/dataflow/support.

I am using service account with all required IAM roles

Upvotes: 0

Views: 609

Answers (2)

Prajna Rai T
Prajna Rai T

Reputation: 1818

Generally The Dataflow job appears to be stuck because no worker activity has been seen in the last 1h can be caused by too long setup progress. In order to solve this issue you can try to increase worker resources (via --machine_type parameter) to overcome the issue.

For example, While installing several dependencies that required building wheels (pystan, fbprophet) which will take more than an hour on the minimal machine (n1-standard-1 with 1 vCPU and 3.75GB RAM). Using a more powerful instance (n1-standard-4 which has 4 times more resources) will solve the problem.

You can debug this by looking at the worker startup logs in cloud logging. You are likely to see pip issues with installing dependencies.

Upvotes: 1

Bruno Volpato
Bruno Volpato

Reputation: 1428

Do you have any error logs showing that Dataflow Workers are crashing when trying to start?

If not, maybe worker VMs are started but they can't reach the Dataflow service, which is often related to network connectivity.

Please note that by default, Dataflow creates jobs using the network and subnetwork default (please check if it exists on your project), and you can change to a specific one by specifying --subnetwork. Check https://cloud.google.com/dataflow/docs/guides/specifying-networks for more information.

Upvotes: 0

Related Questions