CODEWITHSUNDEEP
pythonapache-sparkpysparkjupyter-notebookvirtualenv
The Great
The Great

Reputation: 7723

Installing find spark in virtual environment

I am using pyenv to create a virtual environment.

My pyenv packages are located under the project bio in /.pyenv/versions/bio/lib/python3.7/site-packages

I installed findspark using below

pip install findspark  #it was installed successfully.

I am able to see the below files in the packages directory.

findspark-1.4.2.dist-info
findspark.py

However, when I launch Jupyter notebook from the pyenv directory, I get an error message

import findspark
findspark.init()

ImportError: No module named findspark

Can you please help me understand why do we get this error despite the pip install being successful?

My which Jupyter returns the below path (both in terminal & in jupyter notebook)

`/home/abcd/.pyenv/shims/jupyter

Upvotes: 0

Views: 6059

Answers (1)

Diptangsu Goswami
Diptangsu Goswami

Reputation: 5975

Jupyter notebook does not get launched from within the virtualenv even though you activated the virtualenv in the terminal session.

If you import sys and print out sys.executable, you'll realise that the first value of the python executable isn't that of the virtualenv.

You need to add the python of the virtualenv as a kernel.

$ virtualenv -p python3 pysparkvenv
$ source pysparkvenv/bin/activate
(pysparkvenv) $ pip install findspark jupiter  # etc.

(pysparkvenv) $ python -m ipykernel install --user

(pysparkvenv) $ which python
/home/disciple/Desktop/sample/pysparkvenv/bin/python

# make the necessary change to the python path
(pysparkvenv) $ sudo /home/disciple/Desktop/sample/pysparkvenv/bin/python -m ipykernel install --name pysparkvenv

This will create a new kernel which will be available in the dropdown list.

You can check if the kernel was created like this

$ jupyter kernelspec list
Available kernels:
  python3        /home/disciple/.local/share/jupyter/kernels/python3
  pysparkvenv    /usr/local/share/jupyter/kernels/pysparkvenv

After this, you can launch jupyter notebook from anywhere and a new kernel will be available. Select this and you'll have all the modules you installed inside the virtualenv.

pysparkvenv kernel

Upvotes: 2

Related Questions