Python Packaging and Distribution Scenario

Question

I am still relatively new to python packaging, each time I think I find "the" solution, I am thrown another curve ball, here is my problem followed by what I've tried:

• I have CentOS and Ubuntu systems with Python 2.7.3 installed that is partitioned from the net so I have to create an "all in one package"
• The target system does NOT have setuptools, easy_install, pip, virtualenv installed (this is the problem I'm trying to solve here)
• The requirements.txt (or setup.py install_dependencies) is fairly heavy (Flask, etc...) for the application (though really, this isn't the problem)

My packaging sophistication has progressed slowly:

For connected systems, I had a really nice process going with

• packaging: python2.7 setup.py sdist
• installation: create a virtualenv, untar the distribution, python setup.py install

For the disconnected system, I've tried a few things. Wheels seem to be appropriate but I can't get to the "final" installation that includes setuptools, easy_install, pip. I am new to wheels so perhaps I am missing something obvious.

I started with these references:

• Python on Wheels, this was super helpful but I could not get my .sh scripts, test data, etc... installed so I am actually using a wheel/sdist hybrid right now
• Wheel, the Docs, again, very helpful but I am stuck on "the final mile of a disconnected system"
• I then figured out I could package virtualenv as a wheel :-) Yay
• I then figured out I could package easy_install as a python program :-) Yay, but it depends on setuptools, boo, I can't find how to get these packaged / installed

Is there a reference around for bootstrapping a system that has Python, is disconnected, but does not have setuptools, pip, wheels, virtualenv? My list of things a person must do to install this simple agent is becoming just way too long :/ I suppose if I can finish the dependency chain there must be a way to latch in a custom script to setup.py to shrink the custom steps back down ...

Answer 1

The pip install --download option that @mac mentioned has been deprecated and removed. Instead the documentation states that the pip download method should be used instead. So the workflow should be:

1. Download the python package or installer using your online machine.
2. Install python using the offline method used by your package manager or the python installer for windows on the offline machine.
3. On the online machine use pip download -r requirements.txt where "requirments.txt" contains the packages you will be needing the proper format
4. Use pip install --find-links=<your-dir-here> <pkgname> to install packages on your offline machine.

Answer 2

Your process will likely vary according to what platform you are targeting, but in general, a typical way to get what you are trying to achieve is to download packages on an online machine, copy them over to the offline one, and then install them from a file rather than from a URL or repository).

A possible workflow for RPM-based distros may be:

1. Install python-pip through binary packages (use rpm or yum-downloadonly, to download the package on an online machine, then copy it over and install it on the offline one with rpm -i python-pip.<whatever-version-and-architecture-you-downloaded>).
2. On your online machine, use pip install --download <pkgname> to download the packages you need.
3. scp or rsync the packages to a given directory X onto your offline machine
4. Use pip install --find-links=<your-dir-here> <pkgname> to install packages on your offline machine.

If you have to replicate the process on many servers, I'd suggest you set up your own repositories behind a firewall. In case of pip, it is very easy, as it's just a matter of telling pip to use a directory as its own index:

$ pip install --no-index --find-links=file:///local/dir/ SomePackage

For RPM or DEB repos is a bit more complicated (but not rocket science!), but possibly also not that necessary, as you really only ought to install python-pip once.