CODEWITHSUNDEEP
jupyter-notebookdaskjupyter-labdask-distributed
user40780
user40780

Reputation: 1920

create dask local cluster and share it among different Jupiter notebook

For Dask on a single machine, I would need a global Memory limit. Since i am running/testing a lot of dask application. I created a separate process

from dask.distributed import Client
client = Client(memory_limit='12GB')

And try to connect to it in another process using

client = Client('127.0.0.1:38719')

However, this line stagnates there.. Did I do anything wrong?

Upvotes: 1

Views: 459

Answers (1)

SultanOrazbayev
SultanOrazbayev

Reputation: 16561

The easiest thing to do is to write the scheduler file that contains connection information:

from dask.distributed import Client
client = Client(memory_limit='12GB')
client.write_scheduler_file("dask_scheduler.json")

In a different notebook, you would use:

from dask.distributed import Client
client = Client(scheduler_file='dask_scheduler.json')

Upvotes: 2

Related Questions