CODEWITHSUNDEEP
pythondaskdask-distributed
user3154748
user3154748

Reputation: 35

Is it possible to modify number of workers/threads in an existing distributed client?

I'm optimizing a TPOT pipeline with dask on my local machine. I expect this to run for 48 hrs or even more.

I started a client with a few cores so I can continue using my machine while it's running in the background.

from dask.distributed import Client
client = Client(n_workers=1, threads_per_worker=6, memory_limit="14GB")

I'm wondering if I could add workers/threads to the client so it has increased computing capacity when I'm sleeping or not using my computer.

Upvotes: 1

Views: 374

Answers (1)

MRocklin
MRocklin

Reputation: 57271

Yes, you might consider using Dask's Adaptive scaling: https://docs.dask.org/en/latest/setup/adaptive.html

client.cluster.adapt(minimum=0, maximum=10)

Upvotes: 1

Related Questions