What exactly should be set in PYTHONPATH?

Question

I'm going through and writing a setup doc for other developers at work for a python project and I've been reading up on the PYTHONPATH environment variable. I'm looking at my current development system and think I have a few things set wrong that is causing my IDE (IntelliJ) to behave incorrectly when looking up the python libraries.

I've looked at the documentation here and here and I'm still unsure of what should actually be in the PYTHONPATH environment variable.

I have PYTHONHOME pointed to C:\Python27.

My current PYTHONPATH is set to PYTHONHOME. Should I also add the directories from sys.path?

UPDATE:

Based on the below information, PYTHONPATH does not need to be set unless there are non-standard libraries that you want python to be able to find by default. For instance, when I install wxPython from the installer it will add its libraries to PYTHONPATH. I do set PYTHONHOME to the root of the python installation so that I can add it to my system PATH environment variable so that I can run python from any where.

Answer 1

You don't have to set either of them. PYTHONPATH can be set to point to additional directories with private libraries in them. PYTHONHOME sets the location of default libraries.

Documentation:

PYTHONHOME
Change the location of the standard Python libraries. By default, the libraries are searched in prefix/lib/pythonversion and exec_prefix/lib/pythonversion, where prefix and exec_prefix are installation-dependent directories, both defaulting to /usr/local.

When PYTHONHOME is set to a single directory, its value replaces both prefix and exec_prefix. To specify different values for these, set PYTHONHOME to prefix:exec_prefix.

PYTHONPATH
Augment the default search path for module files. The format is the same as the shell’s PATH: one or more directory pathnames separated by os.pathsep (e.g. colons on Unix or semicolons on Windows). Non-existent directories are silently ignored.

In addition to normal directories, individual PYTHONPATH entries may refer to zipfiles containing pure Python modules (in either source or compiled form). Extension modules cannot be imported from zipfiles.

The default search path is installation dependent, but generally begins with prefix/lib/pythonversion (see PYTHONHOME above). It is always appended to PYTHONPATH.

An additional directory will be inserted in the search path in front of PYTHONPATH as described above under Interface options. The search path can be manipulated from within a Python program as the variable sys.path.

Answer 2

A little bit less 'ad hoc' solution than directly manipulating the PYTHONPATH exists with the use of the flag -e with the pip command, and allows to seamlessly install local libraries that can be imported, and re-imported to reflect changes made.

In order to be able to import mypackage the same way you do with any other module, the correct approach is to use pip locally:

python -m pip install -e /path_to_package/mypackage/

• python -m ensures you are using the pip package from the same python installation you are currently using.

• -e makes it editable, i/e import mypackage will reload after you make some changes, instead of using the cached one.

mypackage must be an installable package, i/e contain an __init__.py file, and a basic setup.py (or pyproject.toml file for pipenv)

minimal setup.py

from setuptools import find_packages, setup

setup(
    name='mypackage',          # Required
    version='0.0.1',           # Required
    packages=find_packages(),  # Required
)

the package structure must be like this:

mypackage/
    setup.py
    mypackage/    <----- this is a folder inside the other `mypackage/` folder
        __init__.py

or as a tree:

└── python_perso                folder
    └── mypackage                   folder
        ├── mypackage                   folder
        │   └── __init__.py
        └── setup.py

[edit] after installation, the directory will look like this:
(for a package named mypackage)

└── python_perso
    └── mypackage
        ├── mypackage
        │   ├── __init__.py
        │   └── __pycache__
        │       └── __init__.cpython-38.pyc
        ├── mypackage.egg-info
        │   ├── PKG-INFO
        │   ├── SOURCES.txt
        │   ├── dependency_links.txt
        │   └── top_level.txt
        └── setup.py

5 directories, 7 files

Answer 3

Here is what I learned: PYTHONPATH is a directory to add to the Python import search path "sys.path", which is made up of current dir. CWD, PYTHONPATH, standard and shared library, and customer library. For example:

% python3 -c "import sys;print(sys.path)"
['', 
'/home/username/Documents/DjangoTutorial/mySite', 
'/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', 
'/usr/local/lib/python3.6/dist-packages', '/usr/lib/python3/dist-packages']

where the first path '' denotes the current dir., the 2nd path is via

%export PYTHONPATH=/home/username/Documents/DjangoTutorial/mySite 

which can be added to ~/.bashrc to make it permanent, and the rest are Python standard and dynamic shared library plus third-party library such as django.

As said not to mess with PYTHONHOME, even setting it to '' or 'None' will cause python3 shell to stop working:

% export PYTHONHOME=''
% python3
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007f18a44ff740 (most recent call first):
Aborted (core dumped)

Note that if you start a Python script, the CWD will be the script's directory. For example:

username@bud:~/Documents/DjangoTutorial% python3 mySite/manage.py runserver
==== Printing sys.path ====
/home/username/Documents/DjangoTutorial/mySite # CWD is where manage.py resides
/usr/lib/python3.6
/usr/lib/python3.6/lib-dynload
/usr/local/lib/python3.6/dist-packages
/usr/lib/python3/dist-packages

You can also append a path to sys.path at run-time: Suppose you have a file Fibonacci.py in ~/Documents/Python directory:

username@bud:~/Documents/DjangoTutorial% python3 
>>> sys.path.append("/home/username/Documents")
>>> print(sys.path)
['', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', 
'/usr/local/lib/python3.6/dist-packages', '/usr/lib/python3/dist-packages', 
'/home/username/Documents']
>>> from Python import Fibonacci as fibo

or via

% PYTHONPATH=/home/username/Documents:$PYTHONPATH
% python3
>>> print(sys.path)
['', 
'/home/username/Documents', '/home/username/Documents/DjangoTutorial/mySite', 
'/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', 
'/usr/local/lib/python3.6/dist-packages', '/usr/lib/python3/dist-packages']
>>> from Python import Fibonacci as fibo

Answer 4

For most installations, you should not set these variables since they are not needed for Python to run. Python knows where to find its standard library.

The only reason to set PYTHONPATH is to maintain directories of custom Python libraries that you do not want to install in the global default location (i.e., the site-packages directory).

Make sure to read: http://docs.python.org/using/cmdline.html#environment-variables