How to do Python package management?

Question

Coming from a Node.js + npm background, it is really nightmarish trying to understand all the things related to Python package management. After a few hours of research, I've stumbled upon all those keywords:

• easy_install
• virtualenv
• pip
• setuptools
• distutils
• pypi
• wheel
• egg
• site-packages

Can someone help me decipher those terms and put them in historical context? For example, "distutils was the first package manager but it was superseded by X in Y because Z".

I absolutely love Python (the language) but package management seems like a real nightmare to learn for someone who has been using the amazing npm for the last few years.

Answer 1

As of December 2018 the canonical way for distributing python software is following:

• Use setuptools as the distribution library
• Use wheel as the distribution format

$ python setup.py bdist_wheel

• Once you have proper setup.py file, you can build a source tarball that can be distributed

  $ python setup.py sdist

The sdist command creates a tarball under the dist directory of the source tree. The tarball contains all the Python modules that are part of the source tree

• The final step is to export your package somewhere users can install it via pip. That means publishing your project to PyPI.

make sure you have wheels installed in order to have bdist_wheel installed

pip install wheel

Answer 2

Types of Packages
Egg vs Wheel vs Neither. What's meant by neither is that a python package can be installed from its "source" without being packaged as an egg or wheel.

Packaging Utilities
There are several libraries which provide utilities for packaging python applications, including distutils and setuptools. There is already an excellent post on this.

easy_install
Part of setuptools, allows building and installing python packages. Often discouraged in favor of Pip. Designed to make installation of packages easy, doing the chore of downloading and moving them to the correct place for you (hence the name).

Pip
A package manager for python packages, and a replacement for easy_install! See here for some reasons why people prefer it over easy_install. Can do neat things like install a package directly from a git repository or compile C extensions on the target machine. The latter is debatable as to whether or not it's desirable, but nonetheless it's a nice feature to have if you want it.

PyPI
The python package index, where easy_install and Pip search for available packages, by default. Basically a giant online repository of modules that are accepted by the community.

virtualenv
A way of hacking your environment variables to "isolate" an installation of python and it's related modules. Prefers Pip, because Ian Bicking wrote them both. Basically, you use pip to install virtualenv system wide, which then allows you to create python virtual environments, each with their own copy of python, pip, and assorted modules. This lets you have multiple versions of python or install a module just for testing, without mucking up your system-wide python install.

virtualenvwrapper
A really handy shell script that makes creating and tearing down virtual environments easier.

site-packages
One of the supported locations for installing python modules into. Lives someplace like /usr/lib/pythonX.X/site-packages. There are other supported locations, like dist-packages or user specific locations.

What does all this mean for you?
I'd recommend you don't pay any attention to easy_install and just use pip. Please also always use virtualenv. Usually, the only python modules you should install system-wide on your workstation are pip and virtualenv. I've completely ignored eggs and wheels, but if you plan to distribute packages professionally or host them on PyPI, you probably want to investigate those. Also, if you are creating python packages, you will need to learn to write a setup script, with setuptools. My recommendation is to never use distutils.

Some more Reading
A page on python.org about packaging which covers a lot of these topics
Python packaging is a nightmare
A great post that goes against the most common recommendations, including mine!

Answer 3

There's some mixing in the options you are listing:

• virtualenv - is used to create isolated environments

• site-packages - standard location where python packages / libs reside

• pypi - is a repository

• easy_install - is found on the setuptools package

• pip - was written to improve easy_install.

about python eggs