CODEWITHSUNDEEP
pythonvirtualenvpipenv
Mike75
Mike75

Reputation: 524

python requirements: how to get identical versions?

This question is a consequence of the principal solution of requirement specifation:

python django pip update problems: how to get compatible versions?

I try to synchronize the python requirements between a server and the local development systems. Both are Ubuntu 16.04., so this should be a problem. The server gives me a requirement file (after pip-installing django as described in the other quewtion).

But pip-installing the requirements on the local machine gives some errors, like this:

Could not find a version that satisfies the requirement python-    
apt==1.1.0b1 (from -r requirements.txt (line 3)) (from versions: 0.0.0, 
0.7.8) No matching distribution found for python-apt==1.1.0b1 (from -r 
requirements.txt

Is there a way to ensure, that requirements can be met on similar systems without getting such errors caused by version incompatiblities (or of cause alterantively solve the incompatibility problems)?

Do I need virtual environments to solve this? Or even pipenv? Or is there a simple straightforward way to have two systems with compatible python and package environments?

Upvotes: 2

Views: 1493

Answers (2)

TrinitronX
TrinitronX

Reputation: 5233

Problem

This particular module (python-apt) is only available on PyPi with version 0.7.8. However, this release appears to have been a mistake!

One of the developers & Debian package maintainers for python-apt has stated the following:

Aargh, not this whole PyPI thing again. Nobody ever officially uploaded python-apt there. It is tightly coupled with APT, and not supposed to be distributed in any fashion other than via Debian packages.

There is no, and has never been any support for PyPi. And I can say that I have absolutely no interest in duplicating work there. Source: Debian "Deity" Mailing List 2016-11-22 msg#00094

You can install python-apt from apt, we do not provide python-apt on pip. I recently got control over the pypi entry and need to do something with it. I'm not keen on providing python-apt outside of the distro, though (python-apt and apt versions x.y need to match), so I'd rather just get rid of it, so people stop with questions about outdated versions. Source: python-apt#1883451

So at least for this dependency, it appears that we're out of luck when resolving dependencies python-natively via pip + PyPi. Luckily, the upstream project is hosted on the salsa.debian.org GitLab instance, and pip supports git+ SCM urls now, among other options.

Solutions:

Generally, there are many solutions for resolving such a dependency. The concerns you are trying to address are:

  • Where is this package being provided from?
    • System OS package manager
    • Arbitrary Upstream Release URL
    • Development SCM Repo
    • Forked Git Repo with bugfixes
    • etc...
  • What Version will you be installing?
    • Compatibility concerns:
      • Must be >= 2.0.0
      • Forked repo / feature branch with some bugfix
      • API Compatibility: Not greater than 2.x / Semantic Versioning
    • Development:
      • Just use latest & greatest ( bleeding edge )
      • Use a specific version for my platform or system (e.g.: system OS package, my local forked version in a development directory)
    • QA / Testing:
      • Test against a specific version
      • Test against latest (e.g.: nightly builds)
      • Test against a version provided by a system OS release
  • How should the dependency be resolved?
    • "abstract" dependency on pip-module-name + version constraint
      • Allows for some flexibility later on with where to gather (URL / PyPi / Artifactory) and what version is allowed to satisfy the constraint.
      • User can always override these if needed by specifying arguments to pip, installing using a specific virtualenv, etc...
    • "concrete" dependency on a specific URL + package-name + version
      • Extreme case: Locking to a specific URL + version + sha1 / sha256 / sha** with checksum verification to ensure exact location & file integrity.
      • Less flexible, but most assuredly locked to an exact and precise version & source.
  • Are you developing an "application" or a "library" / Python module?
    • Do the dependencies need to be installed via pip using setup.py install_requires = [...] style resolution? (library)
    • Do the dependencies need to be installed by the application installer via pip install -r requirements.txt? (application)
  • How will your project be released?
    • Who will be installing the package, and how will their system be allowed to resolve the dependency?
    • Is this going to be released on PyPi as a library, or elsewhere as an application? (some rules of thumb follow)
      • Generally speaking:
        • A library tends to want to have wide open ended version specifiers
        • An application wants very specific dependencies to ensure stability (lots of dependencies means lots of generally untested version permutations!)
      • Use setup.py to specify the library dependencies for PyPi
      • Use requirements.txt to specify an application's dependencies
    • Is this going to be packaged as a native OS package? (e.g.: .deb, .rpm, .apk, etc...)
      • The native package manager has dependency resolution too... perhaps use this to ensure native compatibility!
    • What other OS platforms will your package support & how will those platforms resolve the dependency?

So, most generally we can see that there are various concerns all relating to the desired specificity of where and how these dependencies get resolved and installed. There is no "one size fits all" solution here... There are Pros and Cons, and only many different solutions that fall somewhere on the spectrum of:

more specific <---------------------> less specific
reliable compatibility                reliable installability
less testing permutations             more (possibly un-vetted) testing permutations
limited platform support              more platform support (when more permutations are well tested)
dependable known configurations       less dependable known configurations                        
less platform tolerant                more tolerant and agnostic of platforms
more OS native                        less OS native

Latest & Greatest from GitLab Upstream Repo

One solution to the python-apt package issue is to use this git+ URL feature in requirements.txt. This is great for development against the upstream version of python-apt from GitLab. To further isolate installation from the system OS provided version of python-apt, a virtualenv or pip install --user may be desired. For example:

requirements.txt:

--index-url https://pypi.python.org/simple/

-e git+https://salsa.debian.org/apt-team/python-apt.git#egg=python-apt
-e .

This can be used by an example project with setup.py containing:

[...SNIP...] # Boilerplate stuff here

setup(

#[...SNIP...] # Other setup() args here

    platforms=['linux'],
    # Reference:
    #  - https://github.com/pypa/interoperability-peps/pull/30/files#r184839487
    # sudo apt install python3-apt apt-rdepends apt
    # os_requires=[
    #     ['python3-apt', type='packagename', target='run', os='ubuntu'],
    #     ['apt-rdepends', type='packagename', target='run', os='ubuntu'],
    #     ['apt', type='packagename', target='run', os='ubuntu']
    #     ['libapt-pkg-dev', type='packagename', target='build', os='ubuntu']
    # ]
    # Build-deps for apt-python via git SCM: sudo apt install libapt-pkg-dev 
    python_requires='>=3.5',
    install_requires=[
        'python-apt (>= 2.0)',
        # rest of your dependencies here
        #[... SNIP ...]
    ],
    package_dir={'': 'lib'},
    scripts=_glob('bin/*'),

#[...SNIP...]

)

Note: os_requires isn't actually supported yet, but is proposed for a PEP. This might help in the future for external dependencies on packages. It would help in situations like this where a python module is not distributed via PyPi / pip, but instead is only provided via apt / .deb packages on the OS.

Set up your virtualenv or use pip3 install --user if you wish, then continue. Using the git+ requirements feature results in the following when running pip3 install -r requirements.txt:

$ pip3 install -r requirements.txt
Looking in indexes: https://pypi.python.org/simple/
Obtaining file:///../example-project (from -r requirements.txt (line 4))
Obtaining python-apt from git+https://salsa.debian.org/apt-team/python-apt.git#egg=python-apt (from -r requirements.txt (line 3))
  Updating ./example-project-venv/src/python-apt clone
  Running command git fetch -q --tags
  Running command git reset --hard -q c97d4159beae2f9cd42d55d3dff9c37f5c69aa44
ERROR: example-project 0.0.1 has requirement python-apt>=2.0, but you'll have python-apt 0.0.0 which is incompatible.
Installing collected packages: python-apt, example-project
  Running setup.py develop for python-apt
  Running setup.py develop for example-project
Successfully installed example-project python-apt

Note: You'll probably want to install runtime & build / setup.py dependencies for python-apt first:

# Runtime deps (e.g.: Ubuntu 20.04 needs python3-apt, <20.04 needs python-apt):
sudo apt install python3-apt apt
# python-apt pip install deps (also for setup.py / development)
sudo apt install libapt-pkg-dev

Alternative: dependency_links (Note: may be deprecated)

If you are developing a library type module, and also wish to use GitLab as source for python-apt you may want to consider using dependency_links in setup.py to provide the git+ or http(s) tarball release URL rather than in requirements.txt. This is helpful to distinguish an "application" python project from a "library" Python module project. It all depends on what your project's install process looks like. (e.g.: do you want to pip install -r requirements.txt, or just pip install example-module, or python[3] setup.py {sdist,bdist,bdist_rpm, etc...}. It could also be helpful to specify a custom URL for a forked version of python-apt. However, this method is likely to be deprecated sometime soon (if not partially already in new versions of pip). You may want to consider other options for future-proofing your dependency specifications such as PEP 508 or pip install --find-links ... instead.

Also, the "application" vs "library" distinction can be important here, as well as the concepts of "abstract" vs. "concrete" dependencies. A quick summary might be:

This split between abstract and concrete is an important one. It was what allows the PyPI mirroring infrastructure to work. It is what allows a company to host their own private package index. It is even what enables you to fork a library to fix a bug or add a feature and use your own fork. Because an abstract dependency is a name and an optional version specifier you can install it from PyPI or from Crate.io, or from your own filesystem. You can fork a library, change the code, and as long as it has the right name and version specifier that library will happily go on using it.

Setuptools has a feature similar to the Go example. It’s called dependency links and it looks like this:


setup(
    # ...
    dependency_links = [
        "http://packages.example.com/snapshots/",
        "http://example2.com/p/bar-1.0.tar.gz",
    ], ) ```

This “feature” of setuptools removes the abstractness of its
dependencies and hardcodes an exact url from which you can fetch the
dependency from. Now very similarly to Go if we want to modify
packages, or simply fetch them from a different server we’ll need to
go in and edit each package in the dependency chain in order to update
the dependency_links.

Source: caremad.io Blog Post: setup.py vs requirements.txt

For this python-apt example, we might use something like this to lock a "concrete dependency" on v2.0.0:


setup(
    # [...SNIP...]
    dependency_links = [
        "https://salsa.debian.org/apt-team/python-apt/-/archive/2.0.0/python-apt-2.0.0.tar.gz#egg=python-apt"
    ],
    # [...SNIP...]
) `

Note: This "mis-feature" was briefly removed, and then brought back given some usefulness in specifying private package dependency URLs. However, currently pip --process-dependency-links flag has been deprecated, so its' usefulness is probably limited to older versions of Python 2 + pip.

Alternative: PEP 508 Syntax

Newer versions of pip now have URL support for PEP 508 syntax. This is probably the most future-proof method for specifying concrete and abstract dependencies with a complex grammar (See PEP 508 for details). Packages can now be specified in many ways, including custom URLs.

For example, to lock python-apt to v2.0.0 with optional sha256 checksum:

setup(
    # [...SNIP...]
    install_requires=[
        'python-apt@https://salsa.debian.org/apt-team/python-apt/-/archive/2.0.0/python-apt-2.0.0.tar.gz#sha256=1ddbd3eb7cbc1ded7e0e8a2dd75219f0c59c7e062c6e6bfd5c8ff6f656c59a4e',
        # [...SNIP...]
    ],
    # [...SNIP...]
)

requirements.txt:

--index-url https://pypi.python.org/simple/

-e .

Then, pip install -r requirements.txt still works without any extra flags:

$ ./example-project-venv/bin/python3 ./example-project-venv/bin/pip3 install -r requirements.txt 
Looking in indexes: https://pypi.python.org/simple/
Obtaining file://./src/pub/example-project (from -r requirements.txt (line 4))
Requirement already satisfied: graph-tools>=1.5 in ./example-project-venv/lib/python3.8/site-packages (from example-project==0.0.1->-r requirements.txt (line 4)) (1.5)
Collecting python-apt@ https://salsa.debian.org/apt-team/python-apt/-/archive/2.0.0/python-apt-2.0.0.tar.gz#sha256=1ddbd3eb7cbc1ded7e0e8a2dd75219f0c59c7e062c6e6bfd5c8ff6f656c59a4e
  Using cached https://salsa.debian.org/apt-team/python-apt/-/archive/2.0.0/python-apt-2.0.0.tar.gz (458 kB)
Building wheels for collected packages: python-apt
  Building wheel for python-apt (setup.py) ... done
  Created wheel for python-apt: filename=python_apt-0.0.0-cp38-cp38-linux_x86_64.whl size=2040980 sha256=79eeb0d1bb9e3c9785acb68f164a3f72a5777539137d180e9ded7558d2547a49
  Stored in directory: ~/.cache/pip/wheels/c4/09/b5/36fc8c9a1ebe8786620db922f1495da200dce187ee7c618993
Successfully built python-apt
Installing collected packages: python-apt, example-project
  Attempting uninstall: example-project
    Found existing installation: example-project 0.0.1
    Uninstalling example-project-0.0.1:
      Successfully uninstalled example-project-0.0.1
  Running setup.py develop for example-project
Successfully installed example-project python-apt-0.0.0

Alternative: pip install --find-links ...

Another alternative method for installing a "concrete dependency" locked to a specific version is to pass --find-links to pip install with a released tarball file. This method might be helpful for explicitly installing a specific version given a release URL. For example, using python-apt v2.0.0:

$ ./example-project-venv/bin/python3 ./example-project-venv/bin/pip3 install --find-links 'https://salsa.debian.org/apt-team/python-apt/-/archive/2.0.0/python-apt-2.0.0.tar.gz' -r requirements.txt 
Looking in indexes: https://pypi.python.org/simple/
Looking in links: https://salsa.debian.org/apt-team/python-apt/-/archive/2.0.0/python-apt-2.0.0.tar.gz
Obtaining file://./example-project (from -r requirements.txt (line 4))
Requirement already satisfied: graph-tools>=1.5 in ./example-project-venv/lib/python3.8/site-packages (from example-project==0.0.1->-r requirements.txt (line 4)) (1.5)
Collecting python-apt>=2.0
  Downloading https://salsa.debian.org/apt-team/python-apt/-/archive/2.0.0/python-apt-2.0.0.tar.gz (458 kB)
     |████████████████████████████████| 458 kB 614 kB/s 
  WARNING: Requested python-apt>=2.0 from https://salsa.debian.org/apt-team/python-apt/-/archive/2.0.0/python-apt-2.0.0.tar.gz (from example-project==0.0.1->-r requirements.txt (line 4)), but installing version 0.0.0
Building wheels for collected packages: python-apt
  Building wheel for python-apt (setup.py) ... done
  Created wheel for python-apt: filename=python_apt-0.0.0-cp38-cp38-linux_x86_64.whl size=2040783 sha256=d0a8f88c04f202e948b9855837140517d9b2bd3cef72e626221614552a476780
  Stored in directory: ~/.cache/pip/wheels/8a/07/e9/b3c3328bac08c030a5b1e754e01e327b62fd26f9baedf07c15
Successfully built python-apt
ERROR: example-project 0.0.1 has requirement python-apt>=2.0, but you'll have python-apt 0.0.0 which is incompatible.
Installing collected packages: python-apt, example-project
  Attempting uninstall: python-apt
    Found existing installation: python-apt 0.0.0
    Uninstalling python-apt-0.0.0:
      Successfully uninstalled python-apt-0.0.0
  Attempting uninstall: example-project
    Found existing installation: example-project 0.0.1
    Uninstalling example-project-0.0.1:
      Successfully uninstalled example-project-0.0.1
  Running setup.py develop for example-project
Successfully installed example-project python-apt-0.0.0

Base System Debian Package

On Debian & Ubuntu, there are two .deb packages you'll see on the various distribution versions: python3-apt and python-apt (for Python2).

These packages are managed by the APT package manager, and thus are installed in the system location: /usr/lib/python3/dist-packages or /usr/lib/python2.7/dist-packages for Python3 & Python2.7 respectively.

This dist-packages path, and other Python packaging conventions are explained well in this post:

The system has installed Python packages in the global dist-packages directory of each Python version and created symbolic links:

/usr/lib/python2.7/dist-packages/numpy

/usr/lib/python3/dist-packages/numpy

ls -ls /usr/include/numpy
#-> ../lib/python2.7/dist-packages/numpy/core/include/numpy

ls -l /usr/include/python2.7/numpy
#->../../lib/python2.7/dist-packages/numpy/core/include/numpy

ls -l /usr/include/python3.5/numpy
#-> ../../lib/python3/dist-packages/numpy/core/include/numpy

Note the good use of dist-packages instead of site-packages which should be reserved for the system Python.

So, if you're looking to use the base-OS system level version of python3-apt then you'd want to make sure that this path is on your sys.path or PYTHONPATH so import apt will work. Whereas, if you wanted to use the site-packages location, or a virtualenv location... those would have to exist on sys.path / PYTHONPATH instead.

Unfortunately, as mentioned before, there is not yet an official way to state a dependency on an OS package providing a certain version of a python module. However, as long as you're managing your python runtime environment's import path(s), you should be able to use the correct version from the OS package that lives in the dist-packages directory.

Upvotes: 4

Samuel GIFFARD
Samuel GIFFARD

Reputation: 842

Indeed, python-apt's latest version is 0.7.8 https://pypi.org/project/python-apt/

If you're 100% sure it's the same package, try to change it in your requirements.txt file.

On the other hand, you could try seeing where is that python-apt on your local.

import apt
print(apt.__file__)  # or print(apt.__path__)

And then, go there and check manually what is that python-apt package exactly.

Upvotes: -1

Related Questions