How to manage the dask-worker, if I have lot many long running tasks.!

Question

Suppose I have 4 node dask cluster in which at node-1, dask-scheduler is running, and at the rest of the nodes dask-workers is running. And I am collectively submitting 5 long running tasks. So what is happening in this case is, 3 tasks are in running mode and other 2 are in waiting state. But I don't want to wait for the dask-worker to be free rather I want to start a new worker as the tasks are long-running.

Is there any way to achieve this in dask.?

Answer 1

See secede in the documentation