Is it practical to put Python under version control?

Question

Where practical, I like to have tools required for a build under version control. The ideal is that a fresh checkout will run on any machine with a minimal set of tools required to be installed first.

Is it practical to have Python under version control?

How about python packages? My naive attempt to use Sphinx without installing it fails due to a dependency on Docutils. Is there a way to do use it without installing?

Answer 1

Yes!

When you want to deliver tools to your non-technical users without requiring them to install anything, you can put python into scm. In theory, you could setup some other tool and make them all install it so you can ensure that every time they sync they run pip install -r requirements.txt and get all dependencies, but that's not always easy. If you're facing resistance to using python ("it's not compiled; everyone will have to install python"), then putting python in scm is the route for you!

If you were using Linux/Unix, you'd probably use the built-in package manager, so I'll assume you're using Windows. (Note: there are package managers for Windows like scoop, choco, and winget -- they're worth a look.)

You have two options for putting python in scm:

Windows embeddable package (64-bit)

Read more about this package on why so many python installers.

If you want minimal python and to manually provide packages (no using pip), this should be a good method. Theoretically, you can set yourself up to install packages too. However, if you want any dev on your team to be able to install packages, this is probably not a good option.

Windows installer (64-bit)

A regular python install. But we'll do things a bit different.

1. Setup your scm ignores for *.pyc and __pycache__
2. Install python to C:\python3
3. I was careful not to run python to avoid generating any temp files so they'd be easier to catch and ignore.
4. Copy C:\python3 into your project: c:\proj\tools\python3
5. Uninstall python from C:\python3
   • this way your machine is just like your users. You can add the python in your project to your PATH or re-install python once you've confirmed everything works.
6. Create a directory to hold your python modules: c:\proj\tools\pymodules
7. Create an empty __init__.py file in that folder: c:\proj\tools\pymodules\__init__.py
8. Create a work.pth file with a relative path so python can find your python modules: c:\proj\tools\python3\site-packages\work.pth

    ../../../pymodules

9. Now commit everything.
10. Run python -m http.server 8000 --directory . or some other python code so you can double check your ignores are correct.

Now you'll be able to use pip to install new packages and commit them to source control. If you create pymodules\work\__init__.py (empty) and pymodules\work\example.py:

print('Hello from example.py')

You can run it:

c:\proj\tools\python3\python.exe -m work.example

And import it:

import work.example

No messing with paths!

However, you likely need to ensure all of your users run python.exe directly and can't assume it's in their PATH. But since you know where it is in their project, that's not likely a problem. You can create one-click batch files that determine the path to python from their path:

%~dp0\..\..\python3\python.exe -m work.example

We've used something like this setup for years at my work and I recently set this up with the above steps on a new project. We have our project in svn and it works pretty well, but dumping this many files into git may be less desirable.

Answer 2

Python, like a number of applications, don't run 'in-place' wherever the project is checked out - they need to add to path and know a bit about the setup. This is why they are either part of the platform (Mac and Linux) or need a full-blown installer (Windows).

With this consideration, it's probably best not to include Python itself in the repository - you've still got to choose the right binary installer for the platform and run the installer. And then, if you update the version in your version repository, you have to upgrade the target systems. After all this, you will almost certainly not have consistent systems - so wrecking the point of having Python in version control in the first place.

Good versioning and dependency management does require keeping specific versions of tools. Setuptools includes easy_install which makes this easy:

easy_install "pytest==2.2.4"

Note the specific version - this can be omitted if you're not too worried about the specific version, or you can specify a minimum:

easy_install "pytest>2.2"

(Note: there are other tools that work similarly, including pip)

By default, you will be loading from Pypi, and this keeps all historic versions for you in it's repository. Unless you are really concerned about a specific version going missing, this is fine. If millions of dollars or lives are on the line, check the tool into your local repository and install it with easy_install (or similar).

I would strongly recommend using the virtualenv project to virtualize your Python environment. Doing so allows you to create a sandbox into which easy_install installs libraries and tools, so isolating you from any other tools accidentally installed on the system. Virtualenv also can manage specific versions of Python.

One other thought: If replicating a specific environment for build/test purposes is the point, then consider using a cloud/OS virtualization approach such as VirtualBox, VMWare or similar. You can run literally identical OS images over many different machines.

Answer 3

Look at virtualenv and buildout for these needs.

virtualenv lets you isolate the Python that is running your project from any other packages installed with that version of Python. So if your project needs Python x.y the only pre-requisite is that you need to ensure that there is a copy of that version of Python available on the system. Any packages you install within the virtualenv are completely isolated from packages installed outside it.

buildout lets you specify package dependencies, so if you need sphinx to build your documentation you just include this in your buildout.cfg file:

parts =
  sphinx

[sphinx]
recipe = collective.recipe.sphinxbuilder

and when you run buildout it will install collective.recipe.sphinxbuilder, download and install docutils and sphinx within the virtualenv, and build your docs.

Your buildout.cfg can include all of the dependencies required to get your system running from an initial checkout of the folder containing the buildout.cfg and a bootstrap.py file. It makes setting up a system for development or final deployment really easy.

Answer 4

In general, your version control should contain your project. It is almost always a bad idea to include your dependencies there as well, unless you have made an explicit policy decision to always static link against them (which for interpreted code involves keeping them in your source tree). It sounds like what you want is Zero install, a launch-time dependency injector. It essentially allows you to get the benefits of static linking (your users don't need to install your dependencies if you can't/don't want to get into every relevant package manager repository) without the drawbacks (your users end up with multiple, potentially out of sync, versions of common dependencies lying around).

Answer 5

No, just like a compiler the python interpreter should be installed system-wide. The same thing applies to tools such as sphinx and docutils (which is most likely installed when installing sphinx via your distribution's package manager).

The same applies to most python packages, especially those that are used by the application itself and available via PyPi.