What is the relationship between virtualenv and pyenv?

Question

I recently learned how to use virtualenv and virtualenvwrapper in my workflow but I've seen pyenv mentioned in a few guides but I can't seem to get an understanding of what pyenv is and how it is different/similar to virtualenv. Is pyenv a better/newer replacement for virtualenv or a complimentary tool? If the latter what does it do differently and how do the two (and virtualenvwrapper if applicable) work together?

Answer 1

I'm frustrated enough by the lack of good explanations for this question for newcomers to post my own answer here, even though I'm not fully certain this is totally accurate and complete. I think that's because this space has had a lot of alternatives over the years and the "recommended" tools have changed so much.

Fundamental Difference: Python binary Vs. Python libraries

At fundamental level, when you want to start running any particular Python project, you need two things: the Python binary (and its built-ins), and some Python third party libraries; but you don't only have one project: that's why you need tools.

Clear Definitions: "Python" Vs. "Environment"

1. "Python". People often say "Python" when they mean more than just the binary, they include the libraries and third party installs as well. To run Python you need the actual binary; a single file, at a location, such as /usr/bin/python3. This also comes with the Python built-in libraries for the same version. These will be located in a directory somewhere like /usr/lib/python3/dist-packages. And it's also necessary to include the pip install in with the binary, as you can't get much further without pip to install your third party libraries.

2. "Environment". Again, the term "environment" is used so loosely no-one knows where it starts or ends or what particular kind of environment is being referred to. As soon as you start building your project one of the first things you're going to do is use pip to download and install third party libraries. These are installed into your local site_packages directory, which has its expected canonical locations, but can be anywhere as long as it's referenced properly so "Python" can find it. The site_packages directory is essentially your "environment".

Requirements: Multiple "Pythons" used with multiple "environments" each

So this is how it falls out.

Which "Python" is it?

You're going to start using more than one "Python". You'll have Python 3.7, Python 3.10, and it can grow. They can be re-used, but you need to install them all, which requires different locations, and then inform the tools in your project which path it is you are using for this project.

Which "Environment" is it?

And then even for one particular Python version, more than one project will use that same "Python", but with separate, different, "environments". One Python 3.10 project may use one version of the requests library, another 3.10 project may need a different version of requests library, and you don't want the installation of one to clobber the installation of the other: you need separate site_packages directories for each project.

Tools: "Python" management tool Vs. "Environment" management tool

This is where I lose some accuracy. I'm not familiar with all the tools, and they all have "env" in their name. Clever, huh? We can understand if they all have "py" in their name; that seems unavoidable. But if a tool is managing the Python installs but not the environment why is it called pyenv?

"Python" management tool

To be honest I haven't used pyenv before. I've managed Python installs just by using the OS package management, or I've just stuck to one version of Python. That came unstuck recently though, as systems can be dependent on a particular version of Python being in a particular place, so I want to use pyenv this time.

Maybe pyenv manages "environments" to some extent as well, but I'm using it to install multiple "Pythons". When I use pyenv (how to do that is out of scope for this answer) I want to install a particular version of Python, with its built-ins, and an associated pip.

There are few operations required for this because I only do this when I need a version of Python I haven't got yet. When I install Python 3.7.9 with pyenv, and then activate it, I can now start that Python version and use it. I can refer to it on the command line as python. I can say python -m pip install ....

But I haven't installed any libraries into my environment yet. Because I will use Python 3.7.9 on multiple projects, each with different environments...

"Environment" management tool

And that's where an environment management tool comes in. This question refers to virtualenv and virtualenvwrapper, but they're a bit outdated now, and this answer is focusing on the differences and reasons for the tools. I now use venv for creating and activating environments. Once I have selected (activate) a particular python I use that Python to create an "environment" that can also be activated. Once I have activated that environment, whenever I use python it will use the version of Python and (pip) that I created the environment with, and whenever I say pip install I will use that particular set of site_packages to install the third party libraries in.

Project Workflow

So this now sets you up to run multiple projects per Python, with multiple Pythons to choose from.

1. Create and / or activate the version of Python that you need, using your "Python management" tool: pyenv.
2. Using the particular activated Python, create and / or activate the environment for the project, using your "Environment management" tool: venv (or virtualenvwrapper).
3. Now every time you run Python it's the same version, and all pip installs reside in this local project only, and do not impact others.
4. And you can turn it off at any time (deactivate) and activate another one instead.
5. And you can have multiple shell sessions on multiple projects at the same time. Each Python and each environment is activated and referenced using the $PATH and other Linux environment variables in that shell session. Your shell prompt usually displays a name for you to be able to see which particular environment is currently activated.

Further Information

If you want more information about venv particularly, which (correct me if I'm wrong) seems to be the de facto standard at current Python versions you can find in-depth explanations on venv.

Answer 2

Short version:

• virtualenv allows you to create local (per-directory), independent python installations by cloning from existing ones
• pyenv allows you to install (build from source) different versions of Python alongside each other; you can then clone them with virtualenv or use pyenv to select which one to run at any given time

Longer version:

Virtualenv allows you to create a custom Python installation e.g. in a subdirectory of your project. This is done by cloning from an existing Python installation somewhere on your system (some files are copied, some are reused/shared to save space). Each of your projects can thus have their own python (or even several) under their respective virtualenv. It is perfectly fine for some/all virtualenvs to even have the same version of python (e.g. 3.8.5) without conflict - they live separately and don't know about each other. If you want to use any of those pythons from shell, you have to activate it (by running a script which will temporarily modify your PATH to ensure that that virtualenv's bin/ directory comes first). From that point, calling python (or pip etc.) will invoke that virtualenv's version until you deactivate it (which restores the PATH). It is also possible to call into a virtualenv Python using its absolute path - this can be useful e.g. when invoking Python from a script.

Pyenv operates on a wider scale than virtualenv. It is used to install (build from source) arbitrary versions of Python (it holds a register of available versions). By default, they're all installed alongside each other under ~/.pyenv, so they're "more global" than virtualenv. Then, it allows you to configure which version of Python to run when you use the python command (without virtualenv). This can be done at a global level or, separately, per directory (by placing a .python-version file in a directory). It's done by prepending pyenv's shim python script to your PATH (permanently, unlike in virtualenv) which then decides which "real" python to invoke. You can even configure pyenv to call into one of your virtualenv pythons (by using the pyenv-virtualenv plugin). You can also duplicate Python versions (by giving them different names) and let them diverge.

Using pyenv can be a convenient way of installing Python for subsequent virtualenv use.

Answer 3

Pyenv and virtualenv are very different tools that work in different ways to do different things:

• Pyenv is a bash extension - will not work on Windows - that intercepts your calls to python, pip, etc., to direct them to one of several of the system python tool-chains. So you always have all the libraries that you have installed in the selected python version available - as such it is good for users who have to switch between different versions of python.

• VirtualEnv, is pure python so works everywhere, it makes a copy of, optionally a specific version of, python and pip local to the activate environment which may or may not include links to the current system tool-chain, if it does not you can install just a known subset of libraries into that environment. As such it is almost certainly much better for testing and deployment as you know exactly which libraries, at which versions, are used and a global change will not impact your module.

venv python > 3.3

Note that from Python 3.3 onward there is a built in implementation of VirtualEnv called venv (with, on some installations a wrapper called pyvenv - this wrapper is deprecated in Python 3.6), which should probably be used in preference. To avoid possible issues with the wrapper it is often a good idea to use it directly by using /path/to/python3 -m venv desired/env/path or you can use the excellent py python selector on windows with py -3 -m venv desired/env/path. It will create the directory specified with desired/env/path configure and populate it appropriately. In general it is very much like using VirtualEnv.

Additional Tools

There are a number of tools that it is worth mentioning, and considering, as they can help with the use of one or more of the above:

• VirtualEnvWrapper Manage and simplify the use and management of VirtualEnv - Cross Platform.
• pyenv-virtualenv, installed by pyenv-installer, which gives PyEnv tools for managing and interfacing to VirtualEnv - with this you can have a base installation that includes more than one version of python and create isolated environments within each of them - Linux/OS-X. Suggested by Johann Visagie
• PyInstaller can take your python code, possibly developed & tested under VirtualEnv, and bundle it up so that it can run one platforms that do not have your version of python installed - Note that it is not a cross compiler you will need a Windows (virtual-)machine to build Windows installs, etc., but it can be handy even where you can be sure that python will be installed but cannot be sure that the version of python and all the libraries will be compatible with your code.