CODEWITHSUNDEEP
kubernetesdaskdistributed
Dave Hirschfeld
Dave Hirschfeld

Reputation: 768

Can a dask.distributed cluster autoscale different workers independently?

I'd like to set up a dask cluster with a number of different types of workers e.g. normal workers, high-memory workers, GPU workers, ...

As I understand it I can manually create the workers and tag them with resources. What I'd like to do is create a cluster, specifying the min/max number of each type of worker and have it autoscale the exact number of each type of worker based on the number of tasks requesting each type of worker/resource.

Is this possible now or is this something which is on the roadmap (issue I can subscribe to)?

Upvotes: 1

Views: 334

Answers (1)

MRocklin
MRocklin

Reputation: 57281

Is this possible now

As of 2020-02-15 this is not a supported behavior

or is this something which is on the roadmap (issue I can subscribe to)?

Dask isn't centrally managed, and so doesn't have a roadmap. I don't know of any issue about this today, but I wouldn't be surprised if one already existed.

Upvotes: 1

Related Questions