CODEWITHSUNDEEP
daskdask-distributed
evilkonrex
evilkonrex

Reputation: 255

Dask worker resources for distributed workers

When you define worker resources (http://distributed.readthedocs.io/en/latest/resources.html) in distributed multi-processing workers, is the resource pool defined for all processes?

For example, on the worker host I am running: 

dask-worker --nprocs 8 --resources HOST=1

Now, if I submit task(s) that require resources={"HOST":1} does this guarantee that only one of the processes on that machine will execute this task at specific moment in time?

Upvotes: 1

Views: 866

Answers (1)

MRocklin
MRocklin

Reputation: 57261

The resources keyword applies evenly to all processes. All workers will get a single HOST resource. In general for mature deployments on institutional clusters I recommend avoiding the --nprocs keyword and instead creating each dask-worker process individually; this is more explicit.

dask-worker scheduler-address:8786 --resources
dask-worker scheduler-address:8786
dask-worker scheduler-address:8786
dask-worker scheduler-address:8786
dask-worker scheduler-address:8786
dask-worker scheduler-address:8786
dask-worker scheduler-address:8786
dask-worker scheduler-address:8786

Upvotes: 1

Related Questions