CODEWITHSUNDEEP
daskdask-distributed
limx0
limx0

Reputation: 57

How can I run a dask-distributed local cluster from the command line?

I would like to do the equivalent of Client(LocalCluster()) from the command line.

When interacting with distributed from Jupyter notebooks, I end up restarting my kernel often and starting a new LocalCluster each time, as well as refreshing my bokeh webpage.

I would much rather have a process running in the background that I could just connect to, is this possible?

Upvotes: 1

Views: 2147

Answers (1)

MRocklin
MRocklin

Reputation: 57251

The relevant doc page here is http://distributed.readthedocs.io/en/latest/setup.html#using-the-command-line

In one terminal, write the following:

$ dask-scheduler

In another terminal, write the following:

$ dask-worker localhost:8786

The defaults are a bit different here. LocalCluster creates N single-threaded workers while dask-worker starts one N-threaded worker. You can change these defaults with the following keywords

$ dask-worker localhost:8786 --nthreads 1 --nprocs 4

Upvotes: 2

Related Questions