CODEWITHSUNDEEP
pythonscipyhpc
Robbie Mallett
Robbie Mallett

Reputation: 183

Can't import scipy after building from source

I need to use some functionality recently added to Scipy master and not included in 1.5.4.

I've built scipy from source on my home PC (Ubuntu 18) and it runs fine.

I now need to deploy my code on my university's HPC cluster (running Redhat Enterprise).

I've executed the following commands to install Scipy from source there:

$ mkdir custom_modules
$ cd custom_modules
$ git clone https://github.com/scipy/scipy.git
$ mv scipy scipy_dev
$ pip install --upgrade pip
$ pip install cython
$ pip install pybind11
$ cd scipy_dev
$ python setup.py build --compiler=intelem --fcompiler=intelem
$ python setup.py install --user
$ export PYTHONPATH=$PYTHONPATH:/home/ucfarm0/custom_modules/scipy_dev

This seems to work fine.

I then try to run a test script from my home directory: ($ python3 test.py):

import scipy
print(scipy.__version__)

I get the following error message

ImportError: Error importing SciPy: you cannot import SciPy while
        being in scipy source directory; please exit the SciPy source
        tree first and relaunch your Python interpreter.

This is served up to me from the scipy __init__ file.

I don't understand why I'm getting this error because I'm not calling the script from the scipy source directory. I imagine it's something to do with my PYTHONPATH pointing there. But if it didn't how would I ever be able to import? And why isn't this a problem on my home PC?

Upvotes: 0

Views: 216

Answers (1)

Robbie Mallett
Robbie Mallett

Reputation: 183

You should never set the system $PYTHONPATH variable to the SciPy root directory.

This is because when you run setup.py, it is set up in your site-packages directory. You can't just run it out of the box (i.e. from wherever you downloaded it to), because the fortran elements to be installed.

The code was working on my home computer because the install had worked successfully and scipy was in site packages, and I'd luckily set PYTHONPATH wrong so didn't get the error messages.

I finally managed to Install SciPy from source on the HPC, instructions below:

Set up a Virtual Env and Install Modules

#######################
Download and Install SciPy
#######################

$ module unload compilers mpi
$ module load compilers/gnu/4.9.2
$ module load python/3.7.4
$ virtualenv inverse_smrt
$ source inverse_smrt/bin/activate
$ cd inverse_smrt
$ git clone https://github.com/scipy/scipy.git
$ pip install --upgrade pip
$ pip install numpy cython pybind11 xarray six pandas matplotlib xlrd

$ module load lapack openblas
$ echo $OPENBLASROOT
-> /shared/ucl/apps/openblas/0.3.7-serial/gnu-4.9.2

$ mv scipy scipy_dev
$ cd scipy_dev
$ cp site.cfg.example site.cfg
$ nano site.cfg

    Modify the [openblas] section to:

    [openblas]
    libraries = openblas
    library_dirs = /shared/ucl/apps/openblas/0.3.7-serial/gnu-4.9.2/lib
    include_dirs = /shared/ucl/apps/openblas/0.3.7-serial/gnu-4.9.2/include
    runtime_library_dirs = /shared/ucl/apps/openblas/0.3.7-serial/gnu-4.9.2/lib

$ pip install .
$ cd ..
$ nano test.py
    Import scipy
    print(scipy.__version__)
$ which python
$ python test.py

Upvotes: 1

Related Questions