CODEWITHSUNDEEP
pythonpathconda
dariober
dariober

Reputation: 9062

Unexpected Python paths in Conda environment

In a Conda environment (base here), I'm surprised by the order of directories in the Python path:

python -c "import sys; print(sys.path)"
['', 
'/export/projects/III-data/wcmp_bioinformatics/db291g/miniconda3/lib/python37.zip',
'/export/projects/III-data/wcmp_bioinformatics/db291g/miniconda3/lib/python3.7',
'/export/projects/III-data/wcmp_bioinformatics/db291g/miniconda3/lib/python3.7/lib-dynload',
'/export/home/db291g/.local/lib/python3.7/site-packages',
'/export/projects/III-data/wcmp_bioinformatics/db291g/miniconda3/lib/python3.7/site-packages']

As you can see, my local non-Conda path:

/export/home/db291g/.local/lib/python3.7/site-packages

comes before the Conda counterpart:

/export/projects/III-data/wcmp_bioinformatics/db291g/miniconda3/lib/python3.7/site-packages

This means that Python packages installed in miniconda3/lib/python3.7/site-packages will be ignored if they are also found in .local/lib/python3.7/site-packages. In fact, in .local/lib/python3.7/site-packages I have numpy v1.20, but in the Conda environment I need v1.19, which is correctly installed, but superseded by v1.20. This seems to defeat the point of using Conda.

Is there something wrong with my configuration or am I missing something here?

Some info:

which python
/export/projects/III-data/wcmp_bioinformatics/db291g/miniconda3/bin/python

python -V
Python 3.7.12

which conda
/export/projects/III-data/wcmp_bioinformatics/db291g/miniconda3/bin/conda

conda --version
conda 4.11.0

Upvotes: 9

Views: 8703

Answers (1)

merv
merv

Reputation: 77098

This is expected behavior (see PEP 370) and partially why Anaconda recommended against user-level package installations.

The site module is responsible for setting the sys.path when Python is initializing. The code in site.py specifically appends the user site prior to appending the prefix site, which is what leads to this prioritization. The motivation according to PEP 370 is that users would have a Python installed at system-level, but want to prioritize packages they install at the user level, hence the user site should load prior to the prefix site.

Options

There are several options for avoiding the user-level site-packages from getting loaded.

1: Environment variable

The environment variable PYTHONNOUSERSITE will toggle loading of user-level site-packages. Namely,

PYTHONNOUSERSITE=1 python -c "import sys; print(sys.path)"

2: Python -s flag

Alternatively, the Python binary has an -s argument to specifically disable user-level site packages.

python -s -c "import sys; print(sys.path)"

3: Remove (and avoid future) user-level installs

The Conda recommendation is to avoid pip install --user altogether, which would be interpreted that one should remove the ~/.local/lib/python* folders from your system.

4: Automated Conda environment variable

Conda Forge package

The Conda Forge package conda-ecosystem-user-package-isolation will automatically set PYTHONNOUSERSITE=1 during environment activation.

If you would like all environments to have such isolation by default, then consider adding this to the create_default_packages configuration list:

conda config --add create_default_packages conda-ecosystem-user-package-isolation

Note that this package also sets R_LIBS_USER="-", isolating any R environments from user-level packages.

Alternative packages

If you want a more granular option, I have also created separate packages that set just the PYTHONNOUSERSITE=1 and PYTHONPATH="" environment variables, which can be installed with:

## set PYTHONNOUSERSITE=1
conda install merv::envvar-pythonnousersite-true

## clear PYTHONPATH
conda install merv::envvar-pythonpath-null

Upvotes: 12

Related Questions