Why do include paths in python2 and python3 differ?

Question

While trying to make my scons based build system as platform independent as possible I was wondering about the following:

Why does python2 return the include path /usr/local/include/python2.7? This path does not contain Python.h and building fails if I rely on that path.

python2

Use sysconfig inside python2:

$ /usr/bin/python2
Python 2.7.13 (default, Nov 23 2017, 15:37:09) 
[GCC 6.3.0 20170406] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_path('include')
'/usr/local/include/python2.7'

Gives /usr/local/include/python2.7. This is an empty folder.

Call python2-config from shell:

$ /usr/bin/python2-config --includes
-I/usr/include/python2.7 -I/usr/include/x86_64-linux-gnu/python2.7

This gives a different path. I was able to find Python.h in /usr/include/python2.7.

python3

Use sysconfig inside python3:

$ /usr/bin/python3
Python 3.5.3 (default, Nov 23 2017, 11:34:05) 
[GCC 6.3.0 20170406] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_path('include')
'/usr/include/python3.5m'

Call python3-config from shell:

/usr/bin/python3-config --includes
-I/usr/include/python3.5m -I/usr/include/python3.5m

The resulting path /usr/include/python3.5m is the same for boath approaches. Python.h is located here.

anaconda

If I use anaconda python (2 or 3) the paths are also consistent (as with python3).

I already found some workarounds such as placing a softlink to usr/include in usr/local/include or just throwing away the local/ in the path, but both do not really look like a nice solution.

edit:

Currently the incorrect include path in python2 makes my build system not as platform independend as I would like. Adding an (optional) environment variable PYTHON_INCLUDE_PATH gives me the possibility to define the correct path, if python2 is used. But a solution where the correct path is always returned would help me a lot (either on python side or using scons-specific functionality). As my build system is based on scons a

Answer 1

Python uses an installation scheme that differs depending on the platform and on the installation options.

https://docs.python.org/2/library/sysconfig.html#installation-paths

There are various schemes including posix_local and posix_prefix which determine the location of various installation directories. It seems that sysconfig does not actually record which scheme was used to install the particular build of Python -- it only has the build information.

So when you call sysconfig.get_path() it guesses the scheme based on the default for the current platform [1]. The Python2.7 sysconfig guesses posix_local while the Python3 sysconfig guesses posix_prefix [2].

It looks like both versions of Python are installed using the posix_prefix scheme, so you can specify that when you call sysconfig.get_path:

$ python -c "import sysconfig; print(sysconfig.get_path('include', 'posix_prefix'))"
/usr/include/python2.7

$ python3 -c "import sysconfig; print(sysconfig.get_path('include', 'posix_prefix'))"
/usr/include/python3.5m

[1] https://github.com/python/cpython/blob/2.7/Lib/sysconfig.py#L169

[2] Run sysconfig as a script:

$ python -m sysconfig | head
Platform: "linux-x86_64"
Python version: "2.7"
Current installation scheme: "posix_local"

$ python3 -m sysconfig | head
Platform: "linux-x86_64"
Python version: "3.5"
Current installation scheme: "posix_prefix"

I can't actually find posix_local in the sysconfig source so not sure where it comes from.

EDIT

I looked into this a bit more and learned that it specific to the Debian/Ubuntu versions of Python; upstream Python doesn't use /usr/local/ or have a posix_local scheme. The Debian packages use a kind of hybrid approach which is the same as posix_prefix with the addition of /usr/local/ for modules.

I haven't found a link the source online, but my local system has this in its Python2.7 sysconfig.py (notice the FIXME):

def _get_default_scheme():
    if os.name == 'posix':
        # the default scheme for posix on Debian/Ubuntu is posix_local
        # FIXME: return dist-packages/posix_prefix only for
        #   is_default_prefix and 'PYTHONUSERBASE' not in os.environ and 'real_prefix' not in sys.__dict__
        # is_default_prefix = not prefix or os.path.normpath(prefix) in ('/usr', '/usr/local')
        return 'posix_local'
    return os.name

The Debian python3 sysconfig.py does away with posix_local and uses the same default as upstream python:

def _get_default_scheme():
    if os.name == 'posix':
        # the default scheme for posix is posix_prefix
        return 'posix_prefix'
    return os.name

You might want to replicate that to be compatible with Mac or Windows:

sysconfig.get_path('include', 'posix_prefix' if os.name == 'posix' else os.name)

https://wiki.debian.org/Python#Deviations_from_upstream
https://www.debian.org/doc/packaging-manuals/python-policy/python.html#paths