CODEWITHSUNDEEP
pysparkamazon-emramazon-sagemaker
Itachi
Itachi

Reputation: 2995

installing python package in sagemaker sparkmagic pyspark notebook

I want to install new libraries in a running kernel (not bootstrapping). I'm able to create a sagemaker notebook, which is connected to a EMR cluster, but installing package is a headache. Unable to install packages on notebook. I've tried several methods like installing packages via terminal in jupyterLab. 

$ conda install numba

The installation seems to be working fine on conda_pytorch_p36 notebook, but the packages are not installed on SparkMagic (pyspark) notebook.

Error code:

An error was encountered:
No module named numba
Traceback (most recent call last):
ImportError: No module named numba

The jupyter magic command also doesn't work only in pyspark notebook

!pip install keras

Error:

An error was encountered:
invalid syntax (<stdin>, line 1)
  File "<stdin>", line 1
    !pip install keras
    ^
SyntaxError: invalid syntax

Based on answer in a github post, neither did this work

from subprocess import call
call("pip install dm-sonnet".split(" "))

Upvotes: 4

Views: 5351

Answers (1)

Arshak
Arshak

Reputation: 726

when you are running $ conda install numba via the terminal in JupyterLab, it's actually succeeding the installation on your local environment. The thing is, when you are using Sparkmagic as your kernal, the code in the cells are always running on the spark cluster, not on the local notebook environment. To run the content of a cell locally you should write %%local in the beginning of the cell. After that everything in that cell will run locally and the installed module will be available. Otherwise you should install the module on the remote spark cluster. Read more here: https://github.com/jupyter-incubator/sparkmagic/blob/master/examples/Pyspark%20Kernel.ipynb

Upvotes: 4

Related Questions