Python: Should I save PyPi packages offline as a backup?

Question

My Python projects heavily depends on PyPi packages.
I want to make sure that: in any time in the future: the packages required by my apps will always be available online on PyPi.
For example:-
I found a project on Github that requires PyQt4.
when I tried to run it on my Linux machine,
it crashed on startup because it can't find PyQt4 package on PyPi.

NB: I know that PyQt4 is deprecated

I searched a lot to find an archive for PyPi that still holds PyQt4 package, but I couldn't find them anywhere.

so I had to rewrite that app to make it work on PyQt5.
I only changed the code related to the UI (ie: PyQt4).
other functions were still working.

so the only problem with that app was that PyQt4 package was removed from PyPi.

---

so, my question is: should I save a backup of the PyPi packages I use ?

Answer 1

Short version:

YES, if you want availability... The next big question is how best to keep a backup version of the dependencies? There are some suggestions at the end of this answer.

Long version:

Your question touches on the concept of "Availability" which is one of the three pillars of Information Assurance (or Information Security). The other two pillars are Confidentiality and Integrity... The CIA triad.

PyPI packages are maintained by the owners of those packages, a project that depends on a package and list it as a dependency must take into account the possibility that the owner of the package will pull the package or a version of the package out of PyPI at any moment.

Important Python packages with many dependencies usually are maintained by foundations or organizations that are more responsible with dealing with downstream dependent packages and projects. However keeping support for old packages is very costly and requires extra effort and usually maintainers set a date for end of support, or publish a package lifecycle where they state when a specific version will be removed from the public PyPI server.

Once that happens, the dependents have to update their code (as you did), or provide the original dependency via alternative means.

This topic is very important for procurement in libraries, universities, laboratories, companies, and government agencies where a software tool might have dependencies on other software packages (or ecosystem), and where "availability" should be addressed adequately.

Addressing this risk might mean anything from ensuring high availability at all costs, to living with the risk of losing one or more dependencies... A risk management approach should be used to make informed choices affecting the "security" of your project.

Also it should be noted that, some packages require binary executable or binary libraries or access to a an online API service, which should also be available for the package to work properly, and that complicates the risk analysis and complicates the activities necessary to address availability.

Now to make sure that dependencies are always available... I quickly compiled the following list. Note that each option has pros and cons. You should evaluate these and other options based on your needs:

1. Store the virtual environment along with the code. Once you create a virtual environment and install the packages you require for the project in that virtual environment, you can keep the virtual environment as part of your repository for example for posterity.
2. Host your own PyPI instance (or mirror) and keep a copy of packages you depend upon hosted on it: https://packaging.python.org/en/latest/guides/hosting-your-own-index/
3. Use an "artifact management tool" such as Artifactory from https://jfrog.com/artifact-management/, where you can not only host python packages but also Docker images, nmap packages, and other kinds of artifacts.
4. Get the source code of all dependencies, and always build from source.
5. Create a Docker image where the project works properly and keep backups of the image.
6. If the package requires an online API service, think about replacing that service or mocking it by one you can control.

Answer 2

I use Nexus Repository Manager OSS: https://help.sonatype.com/en/download.html

You can proxy pypi and host your own pypi internal images, can be set up directly from a docker image.

And you get the speedup and benefit of local pip install

You use global variables and pip parameters to set it up: Create and use a PyPi proxy repository on nexus

And Nexus can serve lots of other images: Docker, Nuget, Gems, Maven etc. So you have it one place. Restapi for everything.

Answer 3

Given that the package files are available on PyPI, you can use pip to download the *.whl files compiled for specific OS via:

pip download --only-binary=:all: package_name

or the source distribution files *.tar.gz via:

pip download --no-binary=:all: package_name

Both should download the package files for all the available platforms when you specify :all:.

Alternatively, if the package files are not directly available on PyPI but on some archive as is the case for PyQt4, you can manually download those files.

Once you have the package files (either *.whl binaries or *.tar.gz), you should be able to install them without any internet connection from your local package files by:

pip install /path/to/local/package.whl # for *.whl files
pip install /path/to/local/package.tar.gz # for source *.tar.gz files

However, if you decide to backup your package files to a network storage location e.g. Google drive, you'd need an internet connection, since in this case, pip needs to retrieve the files from the URL and then install the package via:

pip install https://drive.google.com/drive/home/package_name.tar.gz