CODEWITHSUNDEEP
dask
MRocklin
MRocklin

Reputation: 57301

All tasks assigned to one worker when using Dask in adaptive mode

When using Dask normally things work fine. However, when I use Dask with an adaptive cluster I find that sometimes all the tasks get assigned to a single worker. Why is this?

Upvotes: 1

Views: 130

Answers (1)

MRocklin
MRocklin

Reputation: 57301

This should be considered a usability bug, and it would be reasonable to file an issue about it.

However, to explain what is going on (at least today 2018-08-09) probably what happens is that

  1. Your scheduler first has no tasks and so has no workers assigned to it
  2. You submit a lot of work from a client, the scheduler responds and asks for many workers
  3. The first worker arrives and the scheduler hands it all of the work
  4. Milliseconds later, several other workers arrive. The scheduler then proceeds to load balance between the available workers

Ideally, the load balancing heuristics should handle the situation. There were older versions of Dask where this performed less well, but usually this is fine. I recommend first updating your version of the dask and distributed packages to the newest possible releases and if that doesn't work, report an issue with a minimal example if possible.

Upvotes: 1

Related Questions