CODEWITHSUNDEEP
pythondebianapt
Quentin Stafford-Fraser
Quentin Stafford-Fraser

Reputation: 521

How to query and manage Debian package repositories in Python?

I want to be able to look at local .deb files and at remote repositories and deduce dependencies etc so that I can build my own repositories and partial mirrors (probably by creating config files for reprepro).

The challenge is that many of the command-line tools to help with this (apt-rdepends etc) assume that you're running on the target system and make use of your local apt cache, whereas I'll often be handling stuff for different Ubuntu and Debian distributions from the one I'm currently running on, so I'd like to do this a bit more at arm's length.

The capable but very poorly-documented python-apt packages let me examine .deb files on the local filesystem and pull out dependencies. I'm now wondering if there are similar tools to parse the Packages.gz files from repositories? (It's not too tricky, but I don't want to reinvent the wheel!)

The overall goal is to create and maintain two repositories: one with our own packages in, and a partial mirror of an Ubuntu distribution with some known required packages plus anything that they, or our own ones, depend upon.

Upvotes: 6

Views: 2468

Answers (3)

Arnie97
Arnie97

Reputation: 1069

Query

If you already know the distro version and architecture for the desired Packages.gz, the python-apt-repo library provides some tooling to parse it. The following example snippet was copied from its readme:

from apt_repo import APTSources, APTRepository

url = 'http://archive.ubuntu.com/ubuntu'
components = ['main', 'universe', 'multiverse', 'restricted']
sources = APTSources([
    APTRepository(url, 'xenial', components),
    APTRepository(url, 'xenial-updates', components),
    APTRepository(url, 'xenial-backports', components),
    APTRepository(url, 'xenial-proposed', components)
])

print([(package.package, package.version) for package in sources.get_packages_by_name('docker.io')])
[('docker.io', '1.10.3-0ubuntu6'), ('docker.io', '1.13.1-0ubuntu1~16.04.2'), ('docker.io', '17.03.2-0ubuntu2~16.04.1')]

Manage

The python-apt-repo library works read-only and does not help to build or modify your own Packages.gz. However it's easy to build the Packages.gz index from a bunch of local *.deb files with Debian official tools, so you might not need to write a Python script for that step:

dpkg-scanpackages -m any/path/with/debs | gzip -9c > Packages.gz

In case dpkg-scanpackages was not present in your Debian installation, it's packaged in dpkg-dev, along with other useful tools for building a Debian repository from scratch, such as dpkg-scansources.

References

Upvotes: 1

number5
number5

Reputation: 16453

You might want to check out repoman, at least can borrow some ideas from it.

E.g. to get deb package info like https://github.com/synack/repoman#get-detailed-information-about-a-package-1

[
{
    "SHA1": "cae8b9a4a821237a24b5757566efdc95391090d4",
    "Maintainer": "Jeremy Grosser <[email protected]>",
    "Description": "server-side, HTML-embedded scripting language (meta-package) This package is a meta-package that, when installed, guarantees that you have at least one of the four server-side versions of the PHP5 interpreter installed.  Removing this package won't remove PHP5 from your system, however it may remove other packages that depend on this one. . PHP5 is an HTML-embedded scripting language. Much of its syntax is borrowed from C, Java and Perl with a couple of unique PHP-specific features thrown in. The goal of the language is to allow web developers to write dynamically generated pages quickly. . Homepage: http://www.php.net/",
    "Package": "php5",
    "Section": "web",
    "MD5sum": "0efa615337f0ba59d8369e4469884730",
    "Installed-Size": "20",
    "Filename": "pool/main/p/php5/php5_5.2.9-digg8_all.deb",
    "Priority": "normal",
    "Depends": "libapache2-mod-php5 (>= 5.2.9-digg8) | php5-cgi (>= 5.2.9-digg8), php5-common (>= 5.2.9-digg8)",
    "Version": "5.2.9-digg8",
    "Architecture": "all",
    "SHA256": "26844b968f6b51c1a02fd59c21455bf6ba47e46e47e53c6676af0ee4c2dd799b",
    "Size": "1024"
}

]

the code https://github.com/synack/repoman/blob/master/repoman/repository.py#L187

Upvotes: 0

Helmut Grohne
Helmut Grohne

Reputation: 6788

A very effective way is to create local apt caches for all the relevant distributions. The tool chdist from the devscripts package allows you to create a number of these caches without the need to use root privileges. You can then use the tools you are used to (e.g. apt-rdepends) to query those caches by wrapping them up in chdist. You can even point python-apt at your local cache using the rootdir keyword argument to apt.cache.Cache where you can then resolve dependencies.

Upvotes: 1

Related Questions