CODEWITHSUNDEEP
daskdask-distributeddask-delayeddask-dataframe
jrdzha
jrdzha

Reputation: 191

Dask multiple clients

Is it possible to have multiple clients in dask? For instance, can I have multiple threads running with one client per thread, so that when one thread blocks, the others can continue? In this case, each client would have separate task graphs that don't depend on each other.

As a followup question, if this is possible, then how can I specify where to run a specific task? When I do dd.read_csv, then call compute, how do I know which client and its associated scheduler / workers is executing this?

Upvotes: 1

Views: 1875

Answers (1)

mdurant
mdurant

Reputation: 28683

Is it possible to have multiple clients in dask

yes, this is possible, you could, for instance, be running computations on one cluster and other computations on another, simultaneously

can I have multiple threads running with one client per thread

It is not clients that run your work, but workers, so I'm not sure what you are asking.

when one thread blocks, the others can continue

Clients are largely async, and there are few operations that should block, and it's up to you when you call them.

when call compute, how do I know which client and its associated scheduler / workers is executing this

thing.compute() will use the default client, which will be the most recently-created one. The function dask.distributed.get_client() would fetch the right one for you.

To pick which to use, you can use either of these:

fut = client.compute(thing) 
fut.result() or client.gather(fut)

with client:
    thing.compute()

Upvotes: 4

Related Questions