Reputation: 355
So I have a private repository at docker hub and I am trying to download image (blobs) manually using HTTP API.
Now there are some issues
But there is no API in docker HUB api to get list of blobs from a tag and then download it.
There is a docker registry api, but there my username password does not work. What to do?
Upvotes: 2
Views: 11096
Reputation: 265180
For an image, you first pull the manifest and parse that for the list of blobs that you need to pull. All of these API's need the same authorization headers used to list the tags. I'm going to be using regctl from my regclient project to query a local registry, but you could also use curl against Hub which I show below.
$ regctl tag ls localhost:5000/library/alpine
3
3-bkup-20210904
3.10
3.11
3.12
3.13
3.14
3.2
3.3
3.4
3.5
3.6
3.7
3.8
3.9
latest
$ regctl manifest get localhost:5000/library/alpine:latest --format body | jq .
{
"manifests": [
{
"digest": "sha256:14b55f5bb845c7b810283290ce057f175de87838be56f49060e941580032c60c",
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "amd64",
"os": "linux"
},
"size": 528
},
{
"digest": "sha256:40f396779ba29da16f29f780963bd4ad5b7719e3eb5dec04516d583713256aa8",
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "arm",
"os": "linux",
"variant": "v6"
},
"size": 528
},
{
"digest": "sha256:392d9d85dff31e34d756be33579f05ef493cb1b0edccc36a11b3295365553bfd",
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "arm",
"os": "linux",
"variant": "v7"
},
"size": 528
},
{
"digest": "sha256:4fb53f12d2ec18199f16d7c305a12c54cda68cc622484bfc3b7346a44d5024ac",
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "arm64",
"os": "linux",
"variant": "v8"
},
"size": 528
},
{
"digest": "sha256:e8d9cf28250078f08e890a3466efbefda68a8feac03cc4076d3ada3397370d6e",
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "386",
"os": "linux"
},
"size": 528
},
{
"digest": "sha256:d860569a59af627dafee0b0f2b8069e31b07fbdaebe552904dbaec28047ccf64",
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "ppc64le",
"os": "linux"
},
"size": 528
},
{
"digest": "sha256:6640b198347e5bf1e9a9dc5fc864e927154275dc31f3d26193b74350a5c94c9f",
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "s390x",
"os": "linux"
},
"size": 528
}
],
"mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
"schemaVersion": 2
}
$ regctl manifest get localhost:5000/library/alpine@sha256:14b55f5bb845c7b810283290ce057f175de87838be56f49060e941580032c60c --format body | jq .
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 1472,
"digest": "sha256:e9adb5357e84d853cc3eb08cd4d3f9bd6cebdb8a67f0415cc884be7b0202416d"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 2812636,
"digest": "sha256:3d243047344378e9b7136d552d48feb7ea8b6fe14ce0990e0cc011d5e369626a"
}
]
}
$ regctl blob get localhost:5000/library/alpine sha256:e9adb5357e84d853cc3eb08cd4d3f9bd6cebdb8a67f0415cc884be7b0202416d | jq .
{
"architecture": "amd64",
"config": {
"Hostname": "",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"/bin/sh"
],
"Image": "sha256:e211ac20c5c7aaa4ed30d5553654d4679082ec48efcb4d164bac6d50d62653fd",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": null
},
"container": "b6ba94212561a8075e1d324fb050db160e25035ffcfbbe5b410e411e2b7000e2",
"container_config": {
"Hostname": "b6ba94212561",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"/bin/sh",
"-c",
"#(nop) ",
"CMD [\"/bin/sh\"]"
],
"Image": "sha256:e211ac20c5c7aaa4ed30d5553654d4679082ec48efcb4d164bac6d50d62653fd",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": {}
},
"created": "2022-03-17T04:01:59.188838147Z",
"docker_version": "20.10.12",
"history": [
{
"created": "2022-03-17T04:01:58.883733237Z",
"created_by": "/bin/sh -c #(nop) ADD file:cf4b631a115c2bbfbd81cad2d3041bceb64a8136aac92ba8a63b6c51d60af764 in / "
},
{
"created": "2022-03-17T04:01:59.188838147Z",
"created_by": "/bin/sh -c #(nop) CMD [\"/bin/sh\"]",
"empty_layer": true
}
],
"os": "linux",
"rootfs": {
"type": "layers",
"diff_ids": [
"sha256:5e03d8cae8773cb694fff1d55da34a40d23c2349087ed15ce68476395d33753c"
]
}
}
$ regctl blob get localhost:5000/library/alpine sha256:3d243047344378e9b7136d552d48feb7ea8b6fe14ce0990e0cc011d5e369626a | tar -tvzf - | head
drwxr-xr-x 0/0 0 2022-03-16 16:15 bin/
lrwxrwxrwx 0/0 0 2022-03-16 16:15 bin/arch -> /bin/busybox
lrwxrwxrwx 0/0 0 2022-03-16 16:15 bin/ash -> /bin/busybox
lrwxrwxrwx 0/0 0 2022-03-16 16:15 bin/base64 -> /bin/busybox
lrwxrwxrwx 0/0 0 2022-03-16 16:15 bin/bbconfig -> /bin/busybox
-rwxr-xr-x 0/0 824984 2022-02-02 13:21 bin/busybox
lrwxrwxrwx 0/0 0 2022-03-16 16:15 bin/cat -> /bin/busybox
lrwxrwxrwx 0/0 0 2022-03-16 16:15 bin/chgrp -> /bin/busybox
lrwxrwxrwx 0/0 0 2022-03-16 16:15 bin/chmod -> /bin/busybox
lrwxrwxrwx 0/0 0 2022-03-16 16:15 bin/chown -> /bin/busybox
...
A few examples if you try doing this with curl:
Get a token (specific to Docker Hub, each registry may have different auth methods and servers):
token=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${repo}:pull" \
| jq -r '.token')
To use a user/password rather than an anonymous login, call curl with the -u user:pass
option:
token=$(curl -u "$user:$pass" -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${repo}:pull" \
| jq -r '.token')
Tags:
curl -H "Authorization: Bearer $token" \
-s "https://registry-1.docker.io/v2/${repo}/tags/list" | jq .
Manifests (be sure to accept all possible manifest media types to avoid the registry converting to legacy formats):
mt_dm="application/vnd.docker.distribution.manifest.v2+json"
mt_dl="application/vnd.docker.distribution.manifest.list.v2+json"
mt_om="application/vnd.oci.image.manifest.v1+json"
mt_oi="application/vnd.oci.image.index.v1+json"
curl -H "Accept: ${mt_dm}" -H "Accept: ${mt_dl}" -H "Accept: ${mt_om}" -H "Accept: ${mt_oi}" \
-H "Authorization: Bearer $token" \
-s "https://registry-1.docker.io/v2/${repo}/manifests/${sha:-$tag}" | jq .
And blobs:
curl -H "Authorization: Bearer $token" \
-s -L -o - "https://registry-1.docker.io/v2/${repo}/blobs/${digest}"
If you want to export everything to be able to later import, regctl image export
will convert the image to a tar of manifests and blobs. This is output in the OCI Layout format, and if you only pull a single image manifest (and not a multi-platform manifest) it will also include the needed files for docker load
to import the tar.
Upvotes: 5
Reputation: 2785
You can manually download a docker image with curl by:
For example, I'll show how you can download the hitch
package from hub.docker.com.
Execute the following command to get an authentication token from Docker Hub within the scope of the 'hitch
' package's namespace.
# get a JSON with an anonymous token
curl -so "token.json" "https://auth.docker.io/token?service=registry.docker.io&scope=<resourcetype>:<component>/<component>:<action>";
# extract token from JSON
token=$(cat token.json | jq -jr ".token")
ā INFO: If you're like me and wondering where the heck is the OCI spec that defines this
/token
endpoint for authentication, know that it doesn't exist š¤¦To learn more about the syntax of this URL and '
scope
' GET variable, see docker's Token Scope Documentation.
The above commands will get a free/temporary token that you can use in subsequent API calls. If all went well, there will be no output from these commands. Here's an example execution
user@disp7456:~$ curl -so "token.json" "https://auth.docker.io/token?service=registry.docker.io&scope=repository:library/hitch:pull";
user@disp7456:~$
user@disp7456:~$ token=$(cat token.json | jq -jr ".token")
user@disp7456:~$
We can download the manifest for the '1.8.0-1
' tag of the 'hitch
' package with the 'GET /v2/<name>/manifests/<reference>
' API endpoint.
curl -o manifest.json -s -H "Authorization: Bearer ${token}" https://registry-1.docker.io/v2/library/<package_name>/manifests/<tag>
And here's an example execution that downloads the manifest for the '1.8.0-1
' tag of the 'hitch
' package
user@disp7456:~$ curl -o manifest.json -s -H "Authorization: Bearer ${token}" https://registry-1.docker.io/v2/library/hitch/manifests/1.8.0-1
user@disp7456:~$
user@disp7456:~$ ls
manifest.json
user@disp7456:~$
The previous step downloaded a file named 'manifest.json
'. This 'manifest.json
' file lists all of the "layers" that make up the image of the 'hitch
' package.
Each "layer" consists of a tarball and some metadata about the layer in json format. The information that we need to download the layer's tar file is located in the 'manifest.json
' file. And the metadata about each layer is also in the 'manifest.json
' file.
The format of the 'manifest.json
' file is 'vnd.docker.distribution.manifest.v1+json
', which is defined in Image Manifest Version 2, Schema 1.
Most importantly, the 'manifest.json
' file contains two parallel arrays of the same length:
Consider this truncated snippet of the manifest for the 'hitch
' package's '1.8.0-1
' tag:
{
"schemaVersion": 1,
"name": "library/hitch",
"tag": "1.8.0-1",
"architecture": "amd64",
"fsLayers": [
{
"blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
},
{
"blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
},
{
"blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
},
{
"blobSum": "sha256:3148f4af0a813bcff0a3ed2562aabfb1b596b52ef36eb5eb4d82ce836350b73a"
},
{
"blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
},
{
"blobSum": "sha256:a0e9543db8c1238572466cf00b55436bc7b7e849f7cb305128f391a94b75c2fc"
},
{
"blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
},
{
"blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
},
{
"blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
},
{
"blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
},
{
"blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
},
{
"blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
},
{
"blobSum": "sha256:728328ac3bde9b85225b1f0d60f5c149f5635a191f5d8eaeeb00e095d36ef9fd"
}
],
"history": [
{
"v1Compatibility": "{\"architecture\":\"amd64\",\"config\":{\"Hostname\":\"\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"ExposedPorts\":{\"443/tcp\":{}},\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[],\"Image\":\"sha256:996009c7d7eb032c9ea750e5decc1f8aedbf4530b892cf4ebc7716e1458f36d9\",\"Volumes\":null,\"WorkingDir\":\"/etc/hitch\",\"Entrypoint\":[\"docker-hitch-entrypoint\"],\"OnBuild\":null,\"Labels\":null},\"container\":\"0ff54ee96c4bbfe77da3b2124720ef95c6154d3bc1d3e40a168920dd818367c4\",\"container_config\":{\"Hostname\":\"0ff54ee96c4b\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"ExposedPorts\":{\"443/tcp\":{}},\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[\"/bin/sh\",\"-c\",\"#(nop) \",\"CMD []\"],\"Image\":\"sha256:996009c7d7eb032c9ea750e5decc1f8aedbf4530b892cf4ebc7716e1458f36d9\",\"Volumes\":null,\"WorkingDir\":\"/etc/hitch\",\"Entrypoint\":[\"docker-hitch-entrypoint\"],\"OnBuild\":null,\"Labels\":{}},\"created\":\"2024-05-14T05:23:11.666992342Z\",\"docker_version\":\"20.10.23\",\"id\":\"6703605aae83084affcafb4abcc7c556f0e436c4992ae224f1f58e88242328cb\",\"os\":\"linux\",\"parent\":\"c48ca3d95161bbcdfcaa2e016a675965d55f4f06147ef4445c69347c5965f188\",\"throwaway\":true}"
},
{
"v1Compatibility": "{\"id\":\"c48ca3d95161bbcdfcaa2e016a675965d55f4f06147ef4445c69347c5965f188\",\"parent\":\"1d2af5a156bbc461d98824c2f6bfe295327d4419105c0b7f88f14cb28d0bb240\",\"created\":\"2024-05-14T05:23:11.581588417Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) EXPOSE 443\"]},\"throwaway\":true}"
},
{
"v1Compatibility": "{\"id\":\"1d2af5a156bbc461d98824c2f6bfe295327d4419105c0b7f88f14cb28d0bb240\",\"parent\":\"8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad\",\"created\":\"2024-05-14T05:23:11.489285564Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ENTRYPOINT [\\\"docker-hitch-entrypoint\\\"]\"]},\"throwaway\":true}"
},
{
"v1Compatibility": "{\"id\":\"8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad\",\"parent\":\"a8d8314458142ee2a4ebccb19f48b6f9c696100103c3d49cbbe7ecd2575120e5\",\"created\":\"2024-05-14T05:23:11.403178706Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) COPY file:1abf3c94dce5dc9f6617dc8d36a6fe6f4f7236189d4819f16cefb54288e80e0d in /usr/local/bin/ \"]}}"
},
{
"v1Compatibility": "{\"id\":\"a8d8314458142ee2a4ebccb19f48b6f9c696100103c3d49cbbe7ecd2575120e5\",\"parent\":\"5a78b0e89bbae2390b83e60174ae1efc583f766eff7dffaffa747ccb67472d0f\",\"created\":\"2024-05-14T05:23:11.304477182Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) WORKDIR /etc/hitch\"]},\"throwaway\":true}"
},
{
"v1Compatibility": "{\"id\":\"5a78b0e89bbae2390b83e60174ae1efc583f766eff7dffaffa747ccb67472d0f\",\"parent\":\"5a12a2c67ff9b5bfad288a4ede18d08c259c301efb85403d08a40ea2ad0eb1f8\",\"created\":\"2024-05-14T05:23:11.160227264Z\",\"container_config\":{\"Cmd\":[\"|5 DISTVER=bullseye PKGCOMMIT=f12ab7958bc4885f3f00311cbca5103d9e6ba794 PKGVER=1 SHASUM=62b3554d668c9d17382415db10898bf661ee76343e4ee364f904457efda6cb1eeee7cb81d7a3897734024812b64b1c0e2dc305605706d81a0c1f6030508bf7e2 SRCVER=1.8.0 /bin/sh -c set -ex; BASE_PKGS=\\\"apt-utils curl dirmngr dpkg-dev debhelper devscripts equivs fakeroot git gnupg pkg-config\\\"; export DEBIAN_FRONTEND=noninteractive; export DEBCONF_NONINTERACTIVE_SEEN=true; tmpdir=\\\"$(mktemp -d)\\\"; cd \\\"$tmpdir\\\"; apt-get update; apt-get install -y --no-install-recommends $BASE_PKGS; git clone https://github.com/varnish/pkg-hitch.git; cd pkg-hitch; git checkout ${PKGCOMMIT}; rm -rf .git; curl -Lf https://hitch-tls.org/source/hitch-${SRCVER}.tar.gz -o $tmpdir/orig.tgz; echo \\\"${SHASUM} $tmpdir/orig.tgz\\\" | sha512sum -c -; tar xavf $tmpdir/orig.tgz --strip 1; sed -i -e \\\"s/@SRCVER@/${SRCVER}/g\\\" -e \\\"s/@PKGVER@/${PKGVER:-1}/g\\\" -e \\\"s/@DISTVER@/$DISTVER/g\\\" debian/changelog; mk-build-deps --install --tool=\\\"apt-get -o Debug::pkgProblemResolver=yes --yes\\\" debian/control; sed -i '' debian/hitch*; dpkg-buildpackage -us -uc -j\\\"$(nproc)\\\"; apt-get -y purge --auto-remove hitch-build-deps $BASE_PKGS; apt-get -y --no-install-recommends install ../*.deb; sed -i 's/daemon = on/daemon = off/' /etc/hitch/hitch.conf; rm -rf /var/lib/apt/lists/* \\\"$tmpdir\\\"\"]}}"
},
{
"v1Compatibility": "{\"id\":\"5a12a2c67ff9b5bfad288a4ede18d08c259c301efb85403d08a40ea2ad0eb1f8\",\"parent\":\"c03ad9230005f64133de4501e14a882ef25f03443da4da55ca002d5619f998be\",\"created\":\"2024-05-14T05:21:33.061082853Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ARG SHASUM=62b3554d668c9d17382415db10898bf661ee76343e4ee364f904457efda6cb1eeee7cb81d7a3897734024812b64b1c0e2dc305605706d81a0c1f6030508bf7e2\"]},\"throwaway\":true}"
},
{
"v1Compatibility": "{\"id\":\"c03ad9230005f64133de4501e14a882ef25f03443da4da55ca002d5619f998be\",\"parent\":\"24e7aee556d6a38bfa2e13430db8a998c023a2920017eabc0b3bf0dd7661bf7d\",\"created\":\"2024-05-14T05:21:32.967727298Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ARG PKGCOMMIT=f12ab7958bc4885f3f00311cbca5103d9e6ba794\"]},\"throwaway\":true}"
},
{
"v1Compatibility": "{\"id\":\"24e7aee556d6a38bfa2e13430db8a998c023a2920017eabc0b3bf0dd7661bf7d\",\"parent\":\"f0d07a99d7d1f0b849a4cbe8fc4552d374f4448c2e7f8bfd908aa43132c4ec34\",\"created\":\"2024-05-14T05:21:32.875807605Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ARG DISTVER=bullseye\"]},\"throwaway\":true}"
},
{
"v1Compatibility": "{\"id\":\"f0d07a99d7d1f0b849a4cbe8fc4552d374f4448c2e7f8bfd908aa43132c4ec34\",\"parent\":\"65c7b6d17437bf7a3216e2fea283071e9b5c0d71c6b97472baa8807a30b5d9d8\",\"created\":\"2024-05-14T05:21:32.781941821Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ARG PKGVER=1\"]},\"throwaway\":true}"
},
{
"v1Compatibility": "{\"id\":\"65c7b6d17437bf7a3216e2fea283071e9b5c0d71c6b97472baa8807a30b5d9d8\",\"parent\":\"863a608d086b1bcf7f9b30ccf57260e6cb5d3d793b4e1131aa8f6041b07a7270\",\"created\":\"2024-05-14T05:21:32.682503634Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ARG SRCVER=1.8.0\"]},\"throwaway\":true}"
},
{
"v1Compatibility": "{\"id\":\"863a608d086b1bcf7f9b30ccf57260e6cb5d3d793b4e1131aa8f6041b07a7270\",\"parent\":\"e00e363f3a25341591a5a5e724e20ae3e70f0396be8483a07c0b39d25d33fecd\",\"created\":\"2024-05-14T01:28:27.043980081Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) CMD [\\\"bash\\\"]\"]},\"throwaway\":true}"
},
{
"v1Compatibility": "{\"id\":\"e00e363f3a25341591a5a5e724e20ae3e70f0396be8483a07c0b39d25d33fecd\",\"created\":\"2024-05-14T01:28:26.699066026Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ADD file:9b38b383dd93169a663eed88edf3f2285b837257ead69dc40ab5ed1fb3f52c35 in / \"]}}"
}
],
...
]
}
The sha256sum used to download the blob of the first layer is found at the first element of the fsLayers[]
array (fsLayers[0]
). The metadata about this first layer is found at the first element of the history[]
array (history[0]
).
The sha256sum used to download the blob of the second layer is found at second element of the fsLayers[]
array (fsLayers[1]
). The metadata about this second layer is found at the second element of the history[]
array (history[1]
).
Et cetera...
So how do we download each of these layers separately, yet organize them such that we can later import them as a single image into docker? The answer to that lies in the Docker Image Specification v1.0.0.
The above spec provides an example tree of the files:
For example, here's what the full archive of
library/busybox
is (displayed intree
format):
.
āāā 5785b62b697b99a5af6cd5d0aabc804d5748abbb6d3d07da5d1d3795f2dcc83e
ā āāā VERSION
ā āāā json
ā āāā layer.tar
āāā a7b8b41220991bfc754d7ad445ad27b7f272ab8b4a2c175b9512b97471d02a8a
ā āāā VERSION
ā āāā json
ā āāā layer.tar
āāā a936027c5ca8bf8f517923169a233e391cbb38469a75de8383b5228dc2d26ceb
ā āāā VERSION
ā āāā json
ā āāā layer.tar
āāā f60c56784b832dd990022afc120b8136ab3da9528094752ae13fe63a2d28dc8c
ā āāā VERSION
ā āāā json
ā āāā layer.tar
āāā repositories
\There are one or more directories named with the ID for each layer in a full image. Each of these directories contains 3 files:
- `VERSION` - The schema version of the `json` file
- `json` - The JSON metadata for an image layer
- `layer.tar` - The Tar archive of the filesystem changeset for an image
layer.The content of the
VERSION
files is simply the semantic version of the JSON metadata schema:
1.0
\And the
repositories
file is another JSON file which describes names/tags:{ "busybox":{ "latest":"5785b62b697b99a5af6cd5d0aabc804d5748abbb6d3d07da5d1d3795f2dcc83e" } }
Every key in this object is the name of a repository, and maps to a collection of tag suffixes. Each tag maps to the ID of the image represented by that tag.
So, as shown in the quote from the spec above, what we need to do is to create a set of directories -- one for each layer -- named after the layer's ID. Each of these layer-specific directories must contain 3 files:
layer.tar
'),json
' -- with no file extension)VERSION
' whose contents is literally just '1.0
'All of the information that we need is in the 'manifest.json
' file that we already downloaded. Let's just loop through each layer that it defines, download the layer.tar
tarball, create the 'json
' metadata file, and hard-code the 'VERSION
' file with the following BASH snippet
num_layers=$(cat manifest.json | jq -r ".history | length")
for ((i = 0 ; i < $num_layers ; i++)); do
layer_blobSum=$(cat manifest.json | jq -r ".fsLayers[$i].blobSum")
layer_metadata=$(cat manifest.json | jq -r ".history[$i].v1Compatibility")
layer_id=$(echo $layer_metadata | jq -r ".id")
echo $layer_id
echo $layer_blobSum
mkdir -p "layers/$layer_id"
echo "1.0" > "layers/$layer_id/VERSION"
echo $layer_metadata > "layers/$layer_id/json"
curl -o "layers/$layer_id/layer.tar" -#LH "Authorization: Bearer ${token}" "https://registry-1.docker.io/v2/library/<package_name>/blobs/${layer_blobSum}"
done
And here's an example execution that executes the above snippet to download all of the layers onto disk in a set of directories as defined by the Docker Image Specification v1.0.0.
user@disp7456:~$ num_layers=$(cat manifest.json | jq -r ".history | length")
user@disp7456:~$
user@disp7456:~$ for ((i = 0 ; i < $num_layers ; i++)); do
layer_blobSum=$(cat manifest.json | jq -r ".fsLayers[$i].blobSum")
layer_metadata=$(cat manifest.json | jq -r ".history[$i].v1Compatibility")
layer_id=$(echo $layer_metadata | jq -r ".id")
echo $layer_id
echo $layer_blobSum
mkdir -p "layers/$layer_id"
echo "1.0" > "layers/$layer_id/VERSION"
echo $layer_metadata > "layers/$layer_id/json"
curl -o "layers/$layer_id/layer.tar" -#LH "Authorization: Bearer ${token}" "https://registry-1.docker.io/v2/library/hitch/blobs/${layer_blobSum}"
done
6703605aae83084affcafb4abcc7c556f0e436c4992ae224f1f58e88242328cb
sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
################################################################ 100.0%
c48ca3d95161bbcdfcaa2e016a675965d55f4f06147ef4445c69347c5965f188
sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
################################################################ 100.0%
1d2af5a156bbc461d98824c2f6bfe295327d4419105c0b7f88f14cb28d0bb240
sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
################################################################ 100.0%
8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad
sha256:3148f4af0a813bcff0a3ed2562aabfb1b596b52ef36eb5eb4d82ce836350b73a
################################################################ 100.0%
a8d8314458142ee2a4ebccb19f48b6f9c696100103c3d49cbbe7ecd2575120e5
sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
################################################################ 100.0%
5a78b0e89bbae2390b83e60174ae1efc583f766eff7dffaffa747ccb67472d0f
sha256:a0e9543db8c1238572466cf00b55436bc7b7e849f7cb305128f391a94b75c2fc
################################################################ 100.0%
5a12a2c67ff9b5bfad288a4ede18d08c259c301efb85403d08a40ea2ad0eb1f8
sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
################################################################ 100.0%
c03ad9230005f64133de4501e14a882ef25f03443da4da55ca002d5619f998be
sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
################################################################ 100.0%
24e7aee556d6a38bfa2e13430db8a998c023a2920017eabc0b3bf0dd7661bf7d
sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
################################################################ 100.0%
f0d07a99d7d1f0b849a4cbe8fc4552d374f4448c2e7f8bfd908aa43132c4ec34
sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
################################################################ 100.0%
65c7b6d17437bf7a3216e2fea283071e9b5c0d71c6b97472baa8807a30b5d9d8
sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
################################################################ 100.0%
863a608d086b1bcf7f9b30ccf57260e6cb5d3d793b4e1131aa8f6041b07a7270
sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
################################################################ 100.0%
e00e363f3a25341591a5a5e724e20ae3e70f0396be8483a07c0b39d25d33fecd
sha256:728328ac3bde9b85225b1f0d60f5c149f5635a191f5d8eaeeb00e095d36ef9fd
################################################################ 100.0%
user@disp7456:~$
user@disp7456:~$ tree layers/
layers/
āāā 1d2af5a156bbc461d98824c2f6bfe295327d4419105c0b7f88f14cb28d0bb240
ā āāā json
ā āāā layer.tar
ā āāā VERSION
āāā 24e7aee556d6a38bfa2e13430db8a998c023a2920017eabc0b3bf0dd7661bf7d
ā āāā json
ā āāā layer.tar
ā āāā VERSION
āāā 5a12a2c67ff9b5bfad288a4ede18d08c259c301efb85403d08a40ea2ad0eb1f8
ā āāā json
ā āāā layer.tar
ā āāā VERSION
āāā 5a78b0e89bbae2390b83e60174ae1efc583f766eff7dffaffa747ccb67472d0f
ā āāā json
ā āāā layer.tar
ā āāā VERSION
āāā 65c7b6d17437bf7a3216e2fea283071e9b5c0d71c6b97472baa8807a30b5d9d8
ā āāā json
ā āāā layer.tar
ā āāā VERSION
āāā 6703605aae83084affcafb4abcc7c556f0e436c4992ae224f1f58e88242328cb
ā āāā json
ā āāā layer.tar
ā āāā VERSION
āāā 863a608d086b1bcf7f9b30ccf57260e6cb5d3d793b4e1131aa8f6041b07a7270
ā āāā json
ā āāā layer.tar
ā āāā VERSION
āāā 8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad
ā āāā json
ā āāā layer.tar
ā āāā VERSION
āāā a8d8314458142ee2a4ebccb19f48b6f9c696100103c3d49cbbe7ecd2575120e5
ā āāā json
ā āāā layer.tar
ā āāā VERSION
āāā c03ad9230005f64133de4501e14a882ef25f03443da4da55ca002d5619f998be
ā āāā json
ā āāā layer.tar
ā āāā VERSION
āāā c48ca3d95161bbcdfcaa2e016a675965d55f4f06147ef4445c69347c5965f188
ā āāā json
ā āāā layer.tar
ā āāā VERSION
āāā e00e363f3a25341591a5a5e724e20ae3e70f0396be8483a07c0b39d25d33fecd
ā āāā json
ā āāā layer.tar
ā āāā VERSION
āāā f0d07a99d7d1f0b849a4cbe8fc4552d374f4448c2e7f8bfd908aa43132c4ec34
āāā json
āāā layer.tar
āāā VERSION
14 directories, 39 files
user@disp7456:~$
Finally, besides these layer-specific dirs, we need one additional file (named simply 'repository
', with no file extension) at the same height as these dirs. As defined by the Docker Image Specification v1.0.0, this file should state the name & tag of the image, and it points to the first layer of the image.
Note the 0th
item in the history[]
array is the first layer of the image, so we can create this file with the following command
start_image=$(cat manifest.json | jq -r ".history[0].v1Compatibility")
start_image_id=$(echo $start_image | jq -r ".id")
cat > layers/repositories <<EOF
{
"<image_name>": { "stable": "$start_image_id" }
}
EOF
And here's an example execution to create our 'repository
' file for the 'hitch
' package.
user@disp7456:~$ start_image=$(cat manifest.json | jq -r ".history[0].v1Compatibility")
user@disp7456:~$
user@disp7456:~$ start_image_id=$(echo $start_image | jq -r ".id")
user@disp7456:~$
user@disp7456:~$ cat > layers/repositories <<EOF
{
"hitch": { "stable": "$start_image_id" }
}
EOF
user@disp7456:~$
user@disp7456:~$ ls layers
1d2af5a156bbc461d98824c2f6bfe295327d4419105c0b7f88f14cb28d0bb240
24e7aee556d6a38bfa2e13430db8a998c023a2920017eabc0b3bf0dd7661bf7d
5a12a2c67ff9b5bfad288a4ede18d08c259c301efb85403d08a40ea2ad0eb1f8
5a78b0e89bbae2390b83e60174ae1efc583f766eff7dffaffa747ccb67472d0f
65c7b6d17437bf7a3216e2fea283071e9b5c0d71c6b97472baa8807a30b5d9d8
6703605aae83084affcafb4abcc7c556f0e436c4992ae224f1f58e88242328cb
863a608d086b1bcf7f9b30ccf57260e6cb5d3d793b4e1131aa8f6041b07a7270
8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad
a8d8314458142ee2a4ebccb19f48b6f9c696100103c3d49cbbe7ecd2575120e5
c03ad9230005f64133de4501e14a882ef25f03443da4da55ca002d5619f998be
c48ca3d95161bbcdfcaa2e016a675965d55f4f06147ef4445c69347c5965f188
e00e363f3a25341591a5a5e724e20ae3e70f0396be8483a07c0b39d25d33fecd
f0d07a99d7d1f0b849a4cbe8fc4552d374f4448c2e7f8bfd908aa43132c4ec34
repositories
user@disp7456:~$
user@disp7456:~$ cat layers/repositories
{
"hitch": { "stable": "6703605aae83084affcafb4abcc7c556f0e436c4992ae224f1f58e88242328cb" }
}
user@disp7456:~$
Your 'layers/
' directory should now be prepared-to-spec for importing the entire image into docker.
For reference, here's the contents of just one of the layers:
user@disp7456:~$ ls layers/8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad/
json layer.tar VERSION
user@disp7456:~$
user@disp7456:~$ cat layers/8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad/VERSION
1.0
user@disp7456:~$
user@disp7456:~$ cat layers/8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad/json
{"id":"8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad","parent":"a8d8314458142ee2a4ebccb19f48b6f9c696100103c3d49cbbe7ecd2575120e5","created":"2024-05-14T05:23:11.403178706Z","container_config":{"Cmd":["/bin/sh -c #(nop) COPY file:1abf3c94dce5dc9f6617dc8d36a6fe6f4f7236189d4819f16cefb54288e80e0d in /usr/local/bin/ "]}}
user@disp7456:~$
user@disp7456:~$ sha256sum layers/8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad/layer.tar
3148f4af0a813bcff0a3ed2562aabfb1b596b52ef36eb5eb4d82ce836350b73a layers/8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad/layer.tar
user@disp7456:~$
user@disp7456:~$ tar -tvf layers/8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad/layer.tar
drwxr-xr-x 0/0 0 2024-05-12 19:00 usr/
drwxr-xr-x 0/0 0 2024-05-12 19:00 usr/local/
drwxr-xr-x 0/0 0 2024-05-14 00:23 usr/local/bin/
-rwxrwxr-x 0/0 319 2024-05-14 00:21 usr/local/bin/docker-hitch-entrypoint
user@disp7456:~$
Finally, you can load the layers as one image into docker with docker image load
tar -cC layers . | docker image load
Here's an example execution
user@disp7456:~$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
user@disp7456:~$
user@disp7456:~$ tar -cC layers . | docker load
e00e363f3a25: Loading layer [==================================================>] 31.43MB/31.43MB
863a608d086b: Loading layer [==================================================>] 32B/32B
65c7b6d17437: Loading layer [==================================================>] 32B/32B
f0d07a99d7d1: Loading layer [==================================================>] 32B/32B
24e7aee556d6: Loading layer [==================================================>] 32B/32B
c03ad9230005: Loading layer [==================================================>] 32B/32B
5a12a2c67ff9: Loading layer [==================================================>] 32B/32B
5a78b0e89bba: Loading layer [==================================================>] 1.573MB/1.573MB
a8d831445814: Loading layer [==================================================>] 32B/32B
8f914c821cbe: Loading layer [==================================================>] 415B/415B
1d2af5a156bb: Loading layer [==================================================>] 32B/32B
c48ca3d95161: Loading layer [==================================================>] 32B/32B
6703605aae83: Loading layer [==================================================>] 32B/32B
user@disp7456:~$
user@disp7456:~$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
hitch stable f07eadb841be 3 weeks ago 85.1MB
user@disp7456:~$
The image is now available in docker
.
The example above was copied from Manually Downloading Container Images (Docker, Github Packages)
Upvotes: 0
Reputation: 18983
There is a docker registry api, but there my username password does not work. What to do?
It probably depends on how you supply them. I didn't try it with a private repository on Docker Hub, but here's a script that downloads public images from Docker Hub, and images from private registries:
#!/usr/bin/env bash
set -eu
image=$1
creds=${2-}
# https://github.com/moby/moby/blob/v20.10.18/vendor/github.com/docker/distribution/reference/normalize.go#L29-L57
# https://github.com/moby/moby/blob/v20.10.18/vendor/github.com/docker/distribution/reference/normalize.go#L88-L105
registry=${image%%/*}
if [ "$registry" = "$image" ] \
|| { [ "`expr index "$registry" .:`" = 0 ] && [ "$registry" != localhost ]; }; then
registry=docker.io
else
image=${image#*/}
fi
if [ "$registry" = docker.io ] && [ "`expr index "$image" /`" = 0 ]; then
image=library/$image
fi
if [ "`expr index "$image" :`" = 0 ]; then
tag=latest
else
tag=${image#*:}
image=${image%:*}
fi
if [ "$registry" = docker.io ]; then
registry=https://registry-1.docker.io
elif ! [[ "$registry" =~ ^localhost(:[0-9]+)$ ]]; then
registry=https://$registry
fi
r=`curl -sS "$registry/v2/" \
-o /dev/null \
-w '%{http_code}:%header{www-authenticate}'`
http_code=`echo "$r" | cut -d: -f1`
curl_args=(-sS -H 'Accept: application/vnd.docker.distribution.manifest.v2+json')
if [ "$http_code" = 401 ]; then
if [ "$registry" = https://registry-1.docker.io ]; then
header_www_authenticate=`echo "$r" | cut -d: -f2-`
header_www_authenticate=`echo "$header_www_authenticate" | sed -E 's/^Bearer +//'`
split_into_lines() {
sed -Ee :1 -e 's/^(([^",]|"([^"]|\")*")*),/\1\n/; t1'
}
header_www_authenticate=`echo "$header_www_authenticate" | split_into_lines`
extract_value() {
sed -E 's/^[^=]+="(([^"]|\")*)"$/\1/; s/\\(.)/\1/g'
}
realm=$(echo "$header_www_authenticate" | grep '^realm=' | extract_value)
service=$(echo "$header_www_authenticate" | grep '^service=' | extract_value)
scope=repository:$image:pull
token=`curl -sS "$realm?service=$service&scope=$scope" | jq -r .token`
curl_args+=(-H "Authorization: Bearer $token")
else
curl_args+=(-u "$creds")
fi
fi
manifest=`curl "${curl_args[@]}" "$registry/v2/$image/manifests/$tag"`
config_digest=`echo "$manifest" | jq -r .config.digest`
config=`curl "${curl_args[@]}" -L "$registry/v2/$image/blobs/$config_digest"`
layers=`echo "$manifest" | jq -r '.layers[] | .digest'`
echo "$layers" | \
while IFS= read -r digest; do
curl "${curl_args[@]}" -L "$registry/v2/$image/blobs/$digest" | wc -c
done
Usage:
$ ./download-image.sh hello-world
$ ./download-image.sh library/hello-world
$ ./download-image.sh docker.io/library/hello-world
$ ./download-image.sh myregistry.com/hello-world testuser:testpassword
$ ./download-image.sh localhost:5000/hello-world
Or to be more precise it retrieves the image config and the layers. The thing is, what are you going to do with them? You can try to create a tar archive that looks like the one produced by docker save
. Which is what docker-drag
basically does. But ideally for that you should know what exactly docker pull
and docker save
do.
To give you some links to the source code, docker pull
(the server part) more or less starts here:
https://github.com/moby/moby/blob/v20.10.18/api/server/router/image/image.go#L37
https://github.com/moby/moby/blob/v20.10.18/api/server/router/image/image_routes.go#L78
https://github.com/moby/moby/blob/v20.10.18/daemon/images/image_pull.go#L54
https://github.com/moby/moby/blob/v20.10.18/daemon/images/image_pull.go#L130
https://github.com/moby/moby/blob/v20.10.18/distribution/pull.go#L52
Most of the relevant code is in distribution/pull_v2.go
(the high-level part).
The code that does the http requests is in vendor/github.com/docker/distribution/registry/client/auth/session.go
and vendor/github.com/docker/distribution/registry/client/repository.go
.
docker save
:
https://github.com/moby/moby/blob/v20.10.18/api/server/router/image/image.go#L31
https://github.com/moby/moby/blob/v20.10.18/api/server/router/image/image_routes.go#L160
https://github.com/moby/moby/blob/v20.10.18/daemon/images/image_exporter.go#L16
https://github.com/moby/moby/blob/v20.10.18/image/tarexport/save.go#L187
Most of the code is in image/tarexport/save.go
.
But well, apparently docker-drag
's developer didn't concern himself with it, and it (docker-drag
) seems to work.
In case of private Docker Hub repositories you probably need to add -u user:password
to the $realm?service=$service&scope=$scope
request.
A couple of remarks about using the API:
user/name
(e.g. nginxproxy/nginx-proxy
). But with official repositories (that all in fact start with library/
) the first segment may be omitted (library/ruby
-> ruby
). Also there might be just one segment (e.g. in private registries), and occasionaly the first part alone is called a repository (example). Or the second./v2/_catalog
route.Upvotes: 0