CODEWITHSUNDEEP
pythonpipvirtualenvpython-venvvirtual-environment
Ark-kun
Ark-kun

Reputation: 6787

Creating a cheap "overlay" Python virtual environment on top of current one without re-installing packages

Problem:

My program is getting a list of (requirements_123.txt, program_123.py) pairs (actually a list of script lines like pip install a==1 b==2 c==3 && program_123.py).

My program needs to run each program in an isolated virtual environment based on the current environment.

Requirements:

Ideal solution: I set some environment variables for each program, pointing to a new virtual environment dir, and then just execute the script and pip does the right thing.

How can I do this?

What do I mean by "overlay": Python already has some "overlays". There are system packages and user packages. User packages "shadow" the system packages, but non-shadowed system packages are still visible to the programs. When pip installs the packages in the user directory, it does not uninstall the system package version. This is the exact behavior I need. I just need a third overlay layer: "system packages", "user packages", "program packages".

Related questions (but they do not consider the user dir packages, only the virtual environments):

"Cascading" virtual Python environnements Is it possible to create nested virtual environments for python?

P.S.

If pip freeze doesn't even work, you have much larger problems lurking.

There are many reasons why the result of pip freeze > requirements.txt does not work in practice:

I've just checked a default notebook instance in Google Cloud and almost half of the pip freeze list looks like this:

threadpoolctl @ file:///tmp/tmp79xdzxkt/threadpoolctl-2.1.0-py3-none-any.whl
tifffile @ file:///home/conda/feedstock_root/build_artifacts/tifffile_1597357726309/work

Also packages like conda-package-handling are not even on PyPI.

Anyways, this is just one of the many reasons why pip freeze | pip install does not work in practice.

Upvotes: 1

Views: 666

Answers (1)

palotasb
palotasb

Reputation: 4648

You can add a .pth file (a site module feature) to the site packages directory of your derived virtual environment with a line pointing to the site-packages path of your base virtual environment.

In shell, you can do it like this:

# Assumes that the base virtual environment exists, activate it.
. base/bin/activate

# Create the derived virtual environment.
python -m venv ./derived

# Make the derived virtual environment import base's packages too.
base_site_packages="$(python -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])')"
derived_site_packages="$(./derived/bin/python -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])')"
echo "$base_site_packages" > "$derived_site_packages"/_base_packages.pth

base_site_packages is usually base/lib/python<VERSION>/site-packages, the code to get it is taken from https://stackoverflow.com/a/46071447/3063 – same for derived_site_packages.

The packages installed in the base environment will be available in the derived environment. You can verify this by doing pip list in the derived environment.

# Deactivating the base environment is optional,
# meaning that the derived environment can be activated directly too.
deactivate

. ./derived/bin/activate
pip list

To install your custom Python packages and run your script in the custom environment, you don't necessarily need to activate the derived environment. You can call the derived Python environment's pip and python directly and it should just work:

./derived/bin/pip install a==1 b==2 c==3
./derived/bin/python program_123.py

Upvotes: 5

Related Questions