CODEWITHSUNDEEP
daskdask-distributeddask-delayed
TheCodeCache
TheCodeCache

Reputation: 962

Can we create a Dask cluster having multiple CPU machines as well as multiple GPU machines both.?

Can we create a dask-cluster with some CPU and some GPU machines together. If yes, how to control a certain task must run only on CPU machine, or some other type of task should run only on GPU machine, and if not specified, it should pick whichever machine is free.?

does dask support such type of cluster.? what is the command that controls the task to run on a specific CPU/GPU machine.?

Upvotes: 3

Views: 636

Answers (1)

MRocklin
MRocklin

Reputation: 57271

You can specify that a Dask worker has certain abstract resources 

dask-worker scheduler:8786 --resources "GPU=2"
dask-worker scheduler:8786 --resources "GPU=2"
dask-worker scheduler:8786 --resources "MEMORY=100e9"

and that a task consumes those resources during execution. 

processed = [client.submit(process, d, resources={'GPU': 1}) for d in data]
final = client.submit(aggregate, processed, resources={'MEMORY': 70e9})

You can use this to model machines with GPUs. Note that these terms GPU and MEMORY are just abstract terms. They could just as easily been FOO and BAR.

See documentation on worker resources for more information.

Upvotes: 2

Related Questions