CODEWITHSUNDEEP
google-cloud-dataflow
Graham Polley
Graham Polley

Reputation: 14781

Dataflow job failed - "Resource worker_pool_resource failed to shut down"

What does the following error mean?

Jan 12, 2016, 11:07:12 AM
(8e14844346add98d): Workflow failed. Causes: (8e14844346addb8e): Step teardown_resource_global_gce_worker_pool535: Resource worker_pool_resource failed to shut down

It would suggest to me that the worker pool for my Dataflow job could not be shut down, however when when I check VM resources there are no VMs running.

We've only started to get this error since since enabling autoscaling.

With autoscaling enabled, this error appeared 3 times in succession:

Prior to enabling autoscaling we never saw this error e.g:

Upvotes: 0

Views: 105

Answers (1)

Sam McVeety
Sam McVeety

Reputation: 3214

The Dataflow service returns this error when it cannot be sure that the machines were shut down successfully. For example, if the shutdown request returns an unexpected error type, the failure is reported so that users can potentially take action rather than incur costs from idle machines.

In this case, the service is hitting a quota limit when attempting to resize the worker pool. To work around this issue, you can set --maxNumWorkers to a value under your maximum Compute Engine quota to prevent this from happening.

Upvotes: 2

Related Questions