Gitlab-CI fails unsatisfiable constraints python missing

Question

I have a node project in GitLab. I have CI setup to build Docker image and upload it to AWS ECR. Which was working fine 3 days ago.

.gitlab-ci.yml

image: docker:latest

services:
  - docker:dind

stages:
  - build

variables:
  REPOSITORY_URL: 708604513256.dkr.ecr.ap-south-1.amazonaws.com/foo-bar/api:latest

before_script:
  - apk add --no-cache curl jq python py-pip
  - pip install awscli

build:
  stage: build
  only:
    - master
  script:
    - $(aws ecr get-login --no-include-email --region ap-south-1)
    - docker build -t $REPOSITORY_URL .
    - docker push $REPOSITORY_URL

Log

Running with gitlab-runner 13.0.0 (c127439c)
   on docker-auto-scale fa6cab46
Preparing the "docker+machine" executor
 Using Docker executor with image docker:latest ...
 Starting service docker:dind ...
 Pulling docker image docker:dind ...
 Using docker image sha256:ded0f436a6a380f373f1840de86b6887cff86600946490b3a44a890bc3d95657 for docker:dind ...
 Waiting for services to be up and running...
 *** WARNING: Service runner-fa6cab46-project-17319365-concurrent-0-f592aab325d1bb8d-docker-0 probably didn't start properly.
 Health check error:
 service "runner-fa6cab46-project-17319365-concurrent-0-f592aab325d1bb8d-docker-0-wait-for-service" timeout
 Health check container logs:
 Service container logs:
 2020-06-03T12:35:33.145705974Z time="2020-06-03T12:35:33.145584090Z" level=info msg="Starting up"
 2020-06-03T12:35:33.147344480Z time="2020-06-03T12:35:33.147268652Z" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found"
 2020-06-03T12:35:33.147688820Z time="2020-06-03T12:35:33.147629824Z" level=warning msg="[!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting --tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING [!]"
 2020-06-03T12:35:33.148653260Z time="2020-06-03T12:35:33.148586467Z" level=info msg="libcontainerd: started new containerd process" pid=18
 2020-06-03T12:35:33.148776016Z time="2020-06-03T12:35:33.148727154Z" level=info msg="parsed scheme: \"unix\"" module=grpc
 2020-06-03T12:35:33.148826946Z time="2020-06-03T12:35:33.148797744Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
 2020-06-03T12:35:33.148911802Z time="2020-06-03T12:35:33.148873819Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}] <nil>}" module=grpc
 2020-06-03T12:35:33.148958763Z time="2020-06-03T12:35:33.148932476Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
 2020-06-03T12:35:33.200268502Z time="2020-06-03T12:35:33.200146030Z" level=info msg="starting containerd" revision=7ad184331fa3e55e52b890ea95e65ba581ae3429 version=v1.2.13 
 2020-06-03T12:35:33.200692597Z time="2020-06-03T12:35:33.200637628Z" level=info msg="loading plugin "io.containerd.content.v1.content"..." type=io.containerd.content.v1 
 2020-06-03T12:35:33.200867133Z time="2020-06-03T12:35:33.200823919Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.btrfs"..." type=io.containerd.snapshotter.v1 
 2020-06-03T12:35:33.201154379Z time="2020-06-03T12:35:33.201104050Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.btrfs" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" 
 2020-06-03T12:35:33.201216139Z time="2020-06-03T12:35:33.201181022Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.aufs"..." type=io.containerd.snapshotter.v1 
 2020-06-03T12:35:33.205676519Z time="2020-06-03T12:35:33.205599239Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.aufs" error="modprobe aufs failed: "ip: can't find device 'aufs'\nmodprobe: can't change directory to '/lib/modules': No such file or directory\n": exit status 1" 
 2020-06-03T12:35:33.205881591Z time="2020-06-03T12:35:33.205836399Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.native"..." type=io.containerd.snapshotter.v1 
 2020-06-03T12:35:33.206055158Z time="2020-06-03T12:35:33.206013249Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." type=io.containerd.snapshotter.v1 
 2020-06-03T12:35:33.206518393Z time="2020-06-03T12:35:33.206440358Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.zfs"..." type=io.containerd.snapshotter.v1 
 2020-06-03T12:35:33.206807173Z time="2020-06-03T12:35:33.206761104Z" level=info msg="skip loading plugin "io.containerd.snapshotter.v1.zfs"..." type=io.containerd.snapshotter.v1 
 2020-06-03T12:35:33.206870377Z time="2020-06-03T12:35:33.206833950Z" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." type=io.containerd.metadata.v1 
 2020-06-03T12:35:33.206991282Z time="2020-06-03T12:35:33.206951423Z" level=warning msg="could not use snapshotter btrfs in metadata plugin" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" 
 2020-06-03T12:35:33.207038857Z time="2020-06-03T12:35:33.207012167Z" level=warning msg="could not use snapshotter aufs in metadata plugin" error="modprobe aufs failed: "ip: can't find device 'aufs'\nmodprobe: can't change directory to '/lib/modules': No such file or directory\n": exit status 1" 
 2020-06-03T12:35:33.207105105Z time="2020-06-03T12:35:33.207071463Z" level=warning msg="could not use snapshotter zfs in metadata plugin" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" 
 2020-06-03T12:35:33.214130639Z time="2020-06-03T12:35:33.214040730Z" level=info msg="loading plugin "io.containerd.differ.v1.walking"..." type=io.containerd.differ.v1 
 2020-06-03T12:35:33.214248026Z time="2020-06-03T12:35:33.214196991Z" level=info msg="loading plugin "io.containerd.gc.v1.scheduler"..." type=io.containerd.gc.v1 
 2020-06-03T12:35:33.228942452Z time="2020-06-03T12:35:33.221454702Z" level=info msg="loading plugin "io.containerd.service.v1.containers-service"..." type=io.containerd.service.v1 
 2020-06-03T12:35:33.228984856Z time="2020-06-03T12:35:33.221493565Z" level=info msg="loading plugin "io.containerd.service.v1.content-service"..." type=io.containerd.service.v1 
 2020-06-03T12:35:33.228990005Z time="2020-06-03T12:35:33.221505991Z" level=info msg="loading plugin "io.containerd.service.v1.diff-service"..." type=io.containerd.service.v1 
 2020-06-03T12:35:33.228993696Z time="2020-06-03T12:35:33.221519285Z" level=info msg="loading plugin "io.containerd.service.v1.images-service"..." type=io.containerd.service.v1 
 2020-06-03T12:35:33.228997084Z time="2020-06-03T12:35:33.221531861Z" level=info msg="loading plugin "io.containerd.service.v1.leases-service"..." type=io.containerd.service.v1 
 2020-06-03T12:35:33.229000512Z time="2020-06-03T12:35:33.221542620Z" level=info msg="loading plugin "io.containerd.service.v1.namespaces-service"..." type=io.containerd.service.v1 
 2020-06-03T12:35:33.229003761Z time="2020-06-03T12:35:33.221553454Z" level=info msg="loading plugin "io.containerd.service.v1.snapshots-service"..." type=io.containerd.service.v1 
 2020-06-03T12:35:33.229006982Z time="2020-06-03T12:35:33.221564536Z" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..." type=io.containerd.runtime.v1 
 2020-06-03T12:35:33.229010338Z time="2020-06-03T12:35:33.221759106Z" level=info msg="loading plugin "io.containerd.runtime.v2.task"..." type=io.containerd.runtime.v2 
 2020-06-03T12:35:33.229013595Z time="2020-06-03T12:35:33.221856820Z" level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"..." type=io.containerd.monitor.v1 
 2020-06-03T12:35:33.229016741Z time="2020-06-03T12:35:33.222190164Z" level=info msg="loading plugin "io.containerd.service.v1.tasks-service"..." type=io.containerd.service.v1 
 2020-06-03T12:35:33.229019947Z time="2020-06-03T12:35:33.222213414Z" level=info msg="loading plugin "io.containerd.internal.v1.restart"..." type=io.containerd.internal.v1 
 2020-06-03T12:35:33.229032207Z time="2020-06-03T12:35:33.225029858Z" level=info msg="loading plugin "io.containerd.grpc.v1.containers"..." type=io.containerd.grpc.v1 
 2020-06-03T12:35:33.229035721Z time="2020-06-03T12:35:33.225067184Z" level=info msg="loading plugin "io.containerd.grpc.v1.content"..." type=io.containerd.grpc.v1 
 2020-06-03T12:35:33.229052495Z time="2020-06-03T12:35:33.225080743Z" level=info msg="loading plugin "io.containerd.grpc.v1.diff"..." type=io.containerd.grpc.v1 
 2020-06-03T12:35:33.229056388Z time="2020-06-03T12:35:33.225092135Z" level=info msg="loading plugin "io.containerd.grpc.v1.events"..." type=io.containerd.grpc.v1 
 2020-06-03T12:35:33.229059541Z time="2020-06-03T12:35:33.225103233Z" level=info msg="loading plugin "io.containerd.grpc.v1.healthcheck"..." type=io.containerd.grpc.v1 
 2020-06-03T12:35:33.229062675Z time="2020-06-03T12:35:33.225113980Z" level=info msg="loading plugin "io.containerd.grpc.v1.images"..." type=io.containerd.grpc.v1 
 2020-06-03T12:35:33.229065735Z time="2020-06-03T12:35:33.225123860Z" level=info msg="loading plugin "io.containerd.grpc.v1.leases"..." type=io.containerd.grpc.v1 
 2020-06-03T12:35:33.229068798Z time="2020-06-03T12:35:33.225163545Z" level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"..." type=io.containerd.grpc.v1 
 2020-06-03T12:35:33.229071960Z time="2020-06-03T12:35:33.225174404Z" level=info msg="loading plugin "io.containerd.internal.v1.opt"..." type=io.containerd.internal.v1 
 2020-06-03T12:35:33.229075191Z time="2020-06-03T12:35:33.225684688Z" level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..." type=io.containerd.grpc.v1 
 2020-06-03T12:35:33.229078297Z time="2020-06-03T12:35:33.225716450Z" level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." type=io.containerd.grpc.v1 
 2020-06-03T12:35:33.229081446Z time="2020-06-03T12:35:33.225727678Z" level=info msg="loading plugin "io.containerd.grpc.v1.version"..." type=io.containerd.grpc.v1 
 2020-06-03T12:35:33.229084533Z time="2020-06-03T12:35:33.225738133Z" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"..." type=io.containerd.grpc.v1 
 2020-06-03T12:35:33.229087706Z time="2020-06-03T12:35:33.225983911Z" level=info msg=serving... address="/var/run/docker/containerd/containerd-debug.sock" 
 2020-06-03T12:35:33.229090873Z time="2020-06-03T12:35:33.226055956Z" level=info msg=serving... address="/var/run/docker/containerd/containerd.sock" 
 2020-06-03T12:35:33.229094308Z time="2020-06-03T12:35:33.226065898Z" level=info msg="containerd successfully booted in 0.026443s" 
 2020-06-03T12:35:33.243372181Z time="2020-06-03T12:35:33.240058390Z" level=info msg="Setting the storage driver from the $DOCKER_DRIVER environment variable (overlay2)"
 2020-06-03T12:35:33.243390431Z time="2020-06-03T12:35:33.240296002Z" level=info msg="parsed scheme: \"unix\"" module=grpc
 2020-06-03T12:35:33.243394742Z time="2020-06-03T12:35:33.240311103Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
 2020-06-03T12:35:33.243398341Z time="2020-06-03T12:35:33.240336947Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}] <nil>}" module=grpc
 2020-06-03T12:35:33.243411874Z time="2020-06-03T12:35:33.240345801Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
 2020-06-03T12:35:33.281680505Z time="2020-06-03T12:35:33.260046911Z" level=info msg="parsed scheme: \"unix\"" module=grpc
 2020-06-03T12:35:33.281700448Z time="2020-06-03T12:35:33.260069151Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
 2020-06-03T12:35:33.281705205Z time="2020-06-03T12:35:33.260086077Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}] <nil>}" module=grpc
 2020-06-03T12:35:33.281709991Z time="2020-06-03T12:35:33.260095568Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
 2020-06-03T12:35:33.302342647Z time="2020-06-03T12:35:33.298907986Z" level=info msg="Loading containers: start."
 2020-06-03T12:35:33.356170350Z time="2020-06-03T12:35:33.346872010Z" level=warning msg="Running modprobe bridge br_netfilter failed with message: ip: can't find device 'bridge'\nbridge                167936  1 br_netfilter\nstp                    16384  1 bridge\nllc                    16384  2 bridge,stp\nip: can't find device 'br_netfilter'\nbr_netfilter           24576  0 \nbridge                167936  1 br_netfilter\nmodprobe: can't change directory to '/lib/modules': No such file or directory\n, error: exit status 1"
 2020-06-03T12:35:33.459014885Z time="2020-06-03T12:35:33.458902627Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.18.0.0/16. Daemon option --bip can be used to set a preferred IP address"
 2020-06-03T12:35:33.512323955Z time="2020-06-03T12:35:33.511972176Z" level=info msg="Loading containers: done."
 2020-06-03T12:35:33.538197317Z time="2020-06-03T12:35:33.537823654Z" level=info msg="Docker daemon" commit=42e35e61f3 graphdriver(s)=overlay2 version=19.03.11
 2020-06-03T12:35:33.538256362Z time="2020-06-03T12:35:33.537959192Z" level=info msg="Daemon has completed initialization"
 2020-06-03T12:35:33.597277606Z time="2020-06-03T12:35:33.593481285Z" level=info msg="API listen on [::]:2375"
 2020-06-03T12:35:33.597295730Z time="2020-06-03T12:35:33.593560537Z" level=info msg="API listen on /var/run/docker.sock"
 *********
 Pulling docker image docker:latest ...
 Using docker image sha256:0bfe00e7bcd536451161fd64b2fe05f4ff6373ce9dd241e9762b68c7f56ce438 for docker:latest ...
Preparing environment
 Running on runner-fa6cab46-project-17319365-concurrent-0 via runner-fa6cab46-srm-1591186323-a8bad638...
Getting source from Git repository
 $ eval "$CI_PRE_CLONE_SCRIPT"
 Fetching changes with git depth set to 50...
 Initialized empty Git repository in /builds/foo/bar/api/.git/
 Created fresh repository.
 From https://gitlab.com/foo/bar/api
  * [new ref]         refs/pipelines/141504246 -> refs/pipelines/141504246
  * [new branch]      master                   -> origin/master
 Checking out 823e4b7a as master...
 Skipping Git submodules setup
Restoring cache
00:01
Downloading artifacts
Running before_script and script
00:01
 $ apk add --no-cache curl jq python py-pip
 fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
 fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
 ERROR: unsatisfiable constraints:
   python (missing):
     required by: world[python]
Running after_script
00:01
Uploading artifacts for failed job
00:02
 ERROR: Job failed: exit code 1

This same CI script worked on the same code a week ago. Judging by the logs, probably GitLab runner is unable to download/install python properly. Are there any alternative commands to install python and pip in GitLab-CI ? Or I should wait till GitLab fixes the bug?

Answer 1

This is not a GitLab issue. docker:stable removed the python package. Using docker image docker:19.03.8 fixed the problem.

Updated .gitlab-ci.yml

image: docker:19.03.8

services:
  - docker:dind

stages:
  - build

variables:
  REPOSITORY_URL: 708604513256.dkr.ecr.ap-south-1.amazonaws.com/foo-bar/api:latest

before_script:
  - apk add --no-cache curl jq python py-pip
  - pip install awscli

build:
  stage: build
  only:
    - master
  script:
    - $(aws ecr get-login --no-include-email --region ap-south-1)
    - docker build -t $REPOSITORY_URL .
    - docker push $REPOSITORY_URL

note: Although I am not sure if we can install awscli with docker:latest any other way. If that's possible, I think that would be a better solution.

Answer 2

@bopsi just FYI the format is invalid in the actual gitlab ci example you provided. It's correct in your actual response. In the gitlab CI example, it should be:

image: docker:19.03.8

cheers!

Answer 3

This is an unexpected side-effect after docker-library (https://hub.docker.com/_/docker/, https://github.com/docker-library/) developers updated the base image for their latest docker images from Alpine 3.11 to Alpine 3.12 (https://github.com/docker-library/docker/issues/240).

Alpine 3.12 removed the un-versioned python symlink and package alias (https://git.alpinelinux.org/aports/commit/?h=3.12-stable&id=5ad0ec7da1064361cc74d56edf7524960f49ef9b) and we have to use either python2 or python3 to install Python, or pin to a specific docker image version (which is almost always a good idea, instead of using :latest or :stable tags). docker:19.03.10 has python, while docker:19.03.11 and later don't have python, but python2 and python3.