CODEWITHSUNDEEP
pythonjupyter-notebookjupytertornadojupyter-kernel
smagnan
smagnan

Reputation: 1267

Connecting a jupyter notebook to an existing and running ipython kernel

Short version: I have a software running its own jupyter kernel. I want to connect to this running kernel from an external source. This works fine with the --existing option of jupyter console but jupyter notebook doesn't have such a functionality.

My issue is more or less this one: How do I connect a jupyter notebook to an existing IPython kernel? but the package showed there isn't working properly anymore. I did some code changes to it to make it "work" but I am still running into issues.

Context: I have a software with an embedded python3 version. This includes a bunch of c++ extensions made available through this embedded python (in the form of extra built-in modules). These are only available through the embedded python3 started with the software and can be used to interact with the software in various ways (similar to the bpy package in Blender for example). It also includes a bunch of modules(understand numpy, pandas, matplotlib and many others etc...)

To facilitate dev, I am "exposing" this embedded python and all modules as a jupyter kernel. I can successfully connect to it externally using jupyter console --existing kernel-<id>.json and get access to all the modules and use them to interact with the software.

Issue: There is no --existing option for notebooks, or code/script equivalents.

The one solution I found (the one used by the module mentioned here: How do I connect a jupyter notebook to an existing IPython kernel?) consists in having a custom kernel manager class, that you can then set with jupyter notebook --NotebookApp.kernel_manager_class=extipy.ExternalIPythonKernelManager --Session.key='b""'. It works by overwriting start_kernel and as it says: "It spins up a new kernel through the call to super().start_kernel(...) but then turns its attention to the kernel which was started by an external python process. Kernel restarts will restart the useless kernel and leave the existing kernel alone."

This is the core file class ExternalIPythonKernelManager file, but due to, I imagine, jupyter changes, it fails now (super(ExternalIPythonKernelManager, self).start_kernel(**kwargs).result() fails as a couroutine doesn't have a .result()). Here is my version of start_kernel that "works":

    async def start_kernel(self, **kwargs):
        kernel_id = await super(ExternalIPythonKernelManager, self).start_kernel(**kwargs)
        if self._should_use_existing():
            self._attach_to_latest_kernel(kernel_id)
        return kernel_id

But running jupyter notebook --NotebookApp.kernel_manager_class=extipy.ExternalIPythonKernelManager --Session.key='b""' now gets me

[I 14:33:17.997 NotebookApp] Kernel started: cdaf10a1-6af9-4f09-9e6f-c6298a0bb16e, name: python3
[IPKernelApp] ERROR | No such comm target registered: jupyter.widget.control
[IPKernelApp] WARNING | No such comm: 8563966b-82ac-4f35-8f9b-a98fceaf5f64

and when I go to the notebook, my custom Ipython kernel isn't available, only the default "Python 3" kernel.

Questions:

Upvotes: 6

Views: 2290

Answers (2)

Yang Bo
Yang Bo

Reputation: 3728

Another solution is to use jupyverse instead of jupyter notebook, as jupyverse supports kernels.allow_external_kernels and kernels.external_connection_dir options to reuse existing running kernels.

pip install jupyverse[auth,jupyterlab]

STARTUP_CODE="a = 42"

CONNECTION_DIR="$(mktemp -d)"
(printf "%s\n" "$STARTUP_CODE" && sleep infinity) | 
jupyter console --kernel=python3 -f="$CONNECTION_DIR/connection-file.json" &
jupyverse --set kernels.allow_external_kernels=true --set kernels.external_connection_dir="$CONNECTION_DIR" --open-browser

0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
[2024-03-31 21:23:30,474 INFO] Running in development mode
[2024-03-31 21:23:30,488 INFO] Starting application
Warning: Input is not a terminal (fd=0).
Jupyter console 6.6.3

Python 3.11.8 (main, Feb  6 2024, 21:21:21) [GCC 13.2.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.16.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: In [1]: a = 42
[2024-03-31 21:23:31,283 INFO] 
[2024-03-31 21:23:31,283 INFO] To access the server, copy and paste this URL:
[2024-03-31 21:23:31,283 INFO] http://127.0.0.1:8000/?token=ce016c609cfa4d8bbcb8c663cbab0c64
[2024-03-31 21:23:31,283 INFO] 
[2024-03-31 21:23:31,289 INFO] Started server process [45786]
[2024-03-31 21:23:31,289 INFO] Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)

In [2]: ^[[27;1R[2024-03-31 21:23:32,425 INFO] Application started
WARNING: your terminal doesn't support cursor position requests (CPR).
In [2]:

Now visit http://127.0.0.1:8000/?token=ce016c609cfa4d8bbcb8c663cbab0c64, you will find the existing kernel under the KERNELS tab. Right click it and you will be able to create a new notebook that attaches to the existing kernel.

enter image description here

Upvotes: 1

Yang Bo
Yang Bo

Reputation: 3728

As mentioned here, you can reuse existing kernel by using --KernelProvisionerFactory.default_provisioner_name=pyxll-provisioner.

For example:

STARTUP_CODE="a = 42"

export PYXLL_IPYTHON_CONNECTION_FILE="$(mktemp -d)/connection-file.json"

(printf "%s\n" "$STARTUP_CODE" && sleep infinity) | 
jupyter console --kernel=python3 -f="$PYXLL_IPYTHON_CONNECTION_FILE" &
jupyter notebook --KernelProvisionerFactory.default_provisioner_name=pyxll-provisioner --log-level=ERROR

0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
[I 2024-03-31 03:19:56.892 LabApp] JupyterLab extension loaded from /home/nixos/demo/.venv/lib/python3.11/site-packages/jupyterlab
[I 2024-03-31 03:19:56.893 LabApp] JupyterLab application directory is /home/nixos/peftai/.venv/share/jupyter/lab
[I 2024-03-31 03:19:56.893 LabApp] Extension Manager is 'pypi'.
Warning: Input is not a terminal (fd=0).
Jupyter console 6.6.3

Python 3.11.8 (main, Feb  6 2024, 21:21:21) [GCC 13.2.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.16.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: In [1]: a = 42

In [2]: ^[[23;1R[C 2024-03-31 03:19:59.664 ServerApp] 
    
    To access the server, open this file in a browser:
        file:///home/nixos/.local/share/jupyter/runtime/jpserver-93926-open.html
    Or copy and paste one of these URLs:
        http://localhost:8888/tree?token=da6ee2fac54afc8591bb9c2de3039036fb15bd5150002e29
        http://127.0.0.1:8888/tree?token=da6ee2fac54afc8591bb9c2de3039036fb15bd5150002e29
WARNING: your terminal doesn't support cursor position requests (CPR).
In [2]: 0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.

You can see jupyter console executes a = 42 and holds on the kernel, then all subsequential jupyter notebook sessions will reuse the existing kernel, where a is initialized to 42 out-of-the-box.

Jupyter Notebook

Upvotes: 1

Related Questions