Maintaining a truly independent Python Installation

Question

I work on a cluster and the cluster already has a python installation on it.

But, like my relationship with R, I would like to maintain a local independent installation of Python. This to me seems like something the authors never intended python to be.

I installed python with the --prefix tag and later got an error from pip while installing packages on the local installation because the local installation was sourcing packages from the server specific folder /usr/lib/pythonX.Y/dist-packages/. An easy fix would be to recompile my version so that it uses the same USC4 encoding as the system-wide installation.

But I don't want to do this, as I would like to be free of the various uncontrollable variables that plague our cluster OSes which lead to an incident where the entire OS had to be refreshed.

sys.path is the culprit which creates this /usr/lib/pythonX.Y/ path using os.path in the site.py file and later imports it.

How can I permanently remove this path from my python installation? preferably during compilation of Python itself. I would only resort to modifying the site.py after I know that there is no other way for me to do it.

UPDATE:

So now I have moved on to using pyenv. But I don't think this is the optimal solution to the problem. I would really like to know why this express decision was made by the authors of python to source system-wide paths to any local python installation.

UPDATE 12-04-2016:

I ran across Anaconda and now I am happy to say that I found a solution, i.e.

where I can maintain a python installation which if provided with all packages within the local installation will first pick these packages and their dependencies from the local installation and then finally look at the system wide paths for other packages. This answers the first problem which I had

sys.path first defines global systemwide library paths finally followed by the local library paths. But in Anaconda sys.path picks the global systemwide lib paths as the second priority place to look.

import sys

from pprint import pprint as p

    p(sys.path)
    ['',
     '/home/kpal/anaconda2/bin',
     '/usr/lib/pymodules/python2.7',
     '/home/kpal/.local/lib/python2.7/site-packages/cooler-0.3.0-py2.7.egg',
     '/home/kpal/.local/lib/python2.7/site-packages/hiclib-0.0.0-py2.7-linux-x86_64.egg',
     '/home/kpal/.local/lib/python2.7/site-packages/mirnylib-0.0.0-py2.7-linux-x86_64.egg',
     '/home/kpal/anaconda2/lib/python27.zip',
     '/home/kpal/anaconda2/lib/python2.7',
     '/home/kpal/anaconda2/lib/python2.7/plat-linux2',
     '/home/kpal/anaconda2/lib/python2.7/lib-tk',
     '/home/kpal/anaconda2/lib/python2.7/lib-old',
     '/home/kpal/anaconda2/lib/python2.7/lib-dynload',
     '/home/kpal/.local/lib/python2.7/site-packages',
     '/home/kpal/anaconda2/lib/python2.7/site-packages/Sphinx-1.3.5-py2.7.egg',
     '/home/kpal/anaconda2/lib/python2.7/site-packages/setuptools-20.3-py2.7.egg',
     '/home/kpal/anaconda2/lib/python2.7/site-packages',
     '/home/kpal/anaconda2/lib/python2.7/site-packages/IPython/extensions',
     '/home/kpal/.ipython']

Using Anaconda's package manager condas I can install nearly all the packages which I need for my work (Scientific data/number crunching). If not I can just do a python install "package" --user

and that would be installed under /home/user/.local/. If it was never found it means that it wasn't present even under the global definition.

FoldedChromatin · Accepted Answer