CODEWITHSUNDEEP
amazon-ec2jupyterdaskamazon-emr
Aleksey Bilogur
Aleksey Bilogur

Reputation: 3846

Using the dask labextenstion to connect to a remote cluster

I'm interested in running a Dask cluster on EMR and interacting with it from inside of a Jupyter Lab notebook running on a separate EC2 instance (e.g. an EC2 instance not within the cluster and not managed by EMR).

The Dask documentation points to dask-labextension as the tool of choice for this use case. dask-labextension relies on a YAML config file (and/or some environment vars) to understand how to talk to the cluster. However, as far as I can tell, this configuration can only be set to point to a local Dask cluster. In other words, you must be in a Jupyter Lab notebook running on an instance within the cluster (presumably on the master instance?) in order to use this extension.

Is my read correct? Is it not currently possible to use dask-labextension with an external Dask cluster?

Upvotes: 2

Views: 344

Answers (1)

MRocklin
MRocklin

Reputation: 57261

Dask Labextension can talk to any Dask cluster that is visible from where your web client is running. If you can connect to a dashboard in a web browser then you can copy that same address to the Dask-Labextension search bar and it will connect.

Upvotes: 2

Related Questions