CODEWITHSUNDEEP
jupyter-notebookjupytergoogle-cloud-dataproc
Imu
Imu

Reputation: 555

Cannot reopen Jupyter notebooks on Google Cloud Dataproc cluster after stopping cluster

I was using Google Cloud Dataproc to run a Jupyter notebook (following these instructions: https://cloud.google.com/dataproc/docs/tutorials/jupyter-notebook).

I ran a notebook, saved it, and then at some point later on, stopped the cluster (using the GUI). Then later I restarted the cluster and tried to run Jupyter notebook again with the same instructions but in the last step, when I try to open Jupyter in Chrome I get:

"This site can't be reached. The webpage at http://<my-cluster-name>:8123/ might be temporarily down or it may have moved permanently to a new web address. ERR_SOCKS_CONNECTION_FAILED."

Also (I don't know if this helps) in the terminal window where I configured my browser, I have a message:

ERROR:child_thread_impl.cc(762)] Request for unknown Channel-associated interface: ui::mojom::GpuMain  
Google Chrome[695:8548] NSWindow warning: adding an unknown subview: <FullSizeContentView: 0x7fdfd3e291e0>. Break on NSLog to debug.  
Google Chrome[695:8548] Call stack:
(
"+callStackSymbols disabled for performance reasons"
)

In the terminal window where I ssh-ed to my cluster I have these messages:

channel 3: open failed: connect failed: Connection refused  
channel 4: open failed: connect failed: Connection refused  
channel 5: open failed: connect failed: Connection refused    
channel 6: open failed: connect failed: Connection refused   
channel 12: open failed: connect failed: Connection refused   
channel 12: open failed: administratively prohibited: open failed  
channel 13: open failed: administratively prohibited: open failed  
channel 14: open failed: administratively prohibited: open failed  
channel 14: open failed: connect failed: Connection refused  
channel 8: open failed: connect failed: Connection refused

Also, earlier BEFORE I had stopped the cluster I could close the jupyter notebooks, disconnect from the cluster, and reopen the jupyter notebook. I only ran into this problem after I had stopped the cluster. Any ideas what might be going on?

Upvotes: 2

Views: 1871

Answers (2)

jake he
jake he

Reputation: 21

I fixed the problem by connecting to master machine using ssh and created a systemd service(followed dennis-huo's comment above).

  1. go to /usr/lib/systemd/system
  2. sudo su

  3. create a systemd unit file called "jupyter-notebook.service" with content

    [Unit]
Description=Start Jupyter Notebook Server at reboot

[Service]
Type=simple
ExecStart=/opt/conda/bin/jupyter notebook --allow-root  --no-browser

[Install]
WantedBy=multi-user.target

  4. systemctl daemon-reload

  5. systemctl enable jupyter-notebook.service
  6. systemctl start jupyter-notebook.service

Next step will be to include above code into dataproc-initialization-actions. Hope that helps.

Upvotes: 2

Dennis Huo
Dennis Huo

Reputation: 10677

This is because the current initialization action explicitly launches the jupyter notebook service calling launch-jupyter-kernel.sh. Initialization actions aren't the same as GCE startup-scripts in that they don't re-run on startup; the intent normally is that initialization actions need not be idempotent, but instead if they want to restart on startup need to add some init.d/systemd configs to do so explicitly.

For the one-off case, you can just SSH into the master, then do:

sudo su
source /etc/profile.d/conda.sh
nohup jupyter notebook --allow-root --no-browser >> /var/log/jupyter_notebook.log 2>&1 &

If you want this to happen automatically on startup, you can try to put that in a startup script via GCE metadata, though if you're doing that at cluster creation time you'll need to make sure it doesn't collide with the Dataproc initialization action (also, startup scripts might run before the dataproc init action, so the first attempt you might just want to allow to fail silently).

Longer term, we should update the initialization action to add the entry into init.d/systemd so that the init action itself configures auto restart on reboot. There's no one dedicated to this at the moment, but if you or anyone you know is up to the task, contributions are always well appreciated; I filed https://github.com/GoogleCloudPlatform/dataproc-initialization-actions/issues/108 to track this feature.

Upvotes: 2

Related Questions