CODEWITHSUNDEEP
daskdask-distributed
MaxBenChrist
MaxBenChrist

Reputation: 547

Redistribute dask tasks among the cluster

I am abusing dask as a task scheduler for long running tasks with map(, pure=False). So I am not interested in the dask graph, I just use dark as a way to distribute unix commands.

Lets say if have 1000 tasks and they run for a week on a cluster of 30 workers. What I have seen is that if a worker goes down, its tasks get redistributed to the remaining workers.

Sometimes I can free resources from other simulations and I add new workers to the desk cluster. However, those workers then have 0 tasks assigned and idle. They only get new tasks if one of the old workers goes down, then the tasks get redistributed.

So, my question: "Can I manually redistribute and shuffle the tasks on the dask cluster"?

Upvotes: 4

Views: 643

Answers (1)

MRocklin
MRocklin

Reputation: 57271

The scheduler should already be balancing old tasks to new workers. Information on our work-stealing policies live here:

http://distributed.readthedocs.io/en/latest/work-stealing.html

So I'm surprirsed to hear that tasks don't automatically rebalance. If you're able to produce an mcve to reproduce the problem I'd enjoy taking a look.

Upvotes: 1

Related Questions