How can I let the gitlab-ci-runner DinD image cache intermediate images?

Question

I have a Dockerfile that starts with installing the texlive-full package, which is huge and takes a long time. If I docker build it locally, the intermedate image created after installation is cached, and subsequent builds are fast.

However, if I push to my own GitLab install and the GitLab-CI build runner starts, this always seems to start from scratch, redownloading the FROM image, and doing the apt-get install again. This seems like a huge waste to me, so I'm trying to figure out how to get the GitLab DinD image to cache the intermediate images between builds, without luck so far.

I have tried using the --cache-dir and --docker-cache-dir for the gitlab-runner register command, to no avail.

Is this even something the gitlab-runner DinD image is supposed to be able to do?

My .gitlab-ci.yml:

build_job:
    script:
    - docker build --tag=example/foo .

My Dockerfile:

FROM php:5.6-fpm
MAINTAINER Roel Harbers <roel.harbers@example.com>
RUN apt-get update && apt-get install -qq -y --fix-missing --no-install-recommends texlive-full
RUN echo Do other stuff that has to be done every build.

I use GitLab CE 8.4.0 and gitlab/gitlab-runner:latest as runner, started as

docker run -d --name gitlab-runner --restart always \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /usr/local/gitlab-ci-runner/config:/etc/gitlab-runner \
    gitlab/gitlab-runner:latest \
; \

The runner is registered using:

docker exec -it gitlab-runner gitlab-runner register \
    --name foo.example.com \
    --url https://gitlab.example.com/ci \
    --cache-dir /cache/build/ \
    --executor docker \
    --docker-image gitlab/dind:latest \
    --docker-privileged \
    --docker-disable-cache false \
    --docker-cache-dir /cache/docker/ \
; \

This creates the following config.toml:

concurrent = 1
[[runners]]
    name = "foo.example.com"
    url = "https://gitlab.example.com/ci"
    token = "foobarsldkflkdsjfkldsj"
    tls-ca-file = ""
    executor = "docker"
    cache_dir = "/cache/build/"
    [runners.docker]
        image = "gitlab/dind:latest"
        privileged = true
        disable_cache = false
        volumes = ["/cache"]
        cache_dir = "/cache/docker/"

(I have experimented with different values for cache_dir, docker_cache_dir and disable_cache, all with the same result: no caching whatsoever)

Answer 1

For build dependencies that do not change so ofter you can do kinda manual caching with gitlab image registry. In CI script you do not explicitely call docker build but rather wrap it in a shell script

# cat build_dependencies.sh
registry=registry.example.com
project=group/project
imagebase=$registry/$project/linux

docker pull $imagebase/devbase:1.0
if [ $? -ne 0 ]; then
   docker build -f devbase.dockerfile -t $imagebase/devbase:1.0 .
   docker push $imagebase/devbase:1.0
fi
...

and call that script in your CI

  ...
  script:
    - ./build_dependencies.sh

The downside to this is that when your devbase.dockerfile is updated this would get unnoticed by CI, so you need to force build and push of a new image. So for dynamicly changing images this does not work well, but for your use case this seems like a possible way to go.

Answer 2

Currently you cannot cache intermediate layers in GitLab Docker-in-Docker. Altough there are plans to add that (that are mentioned in the link below). What you can do today to speed up your DinD build is to use the overlay filesystem. To do this you need to be running a liunx kernel >=3.18 and make sure you load the overlay kernel module. Then you set this variable in your gitlab-ci.yml:

variables:
  DOCKER_DRIVER: overlay

For more information see this issue and in particular this comment on "The state of optimising Docker Builds!", see the "Using docker executor with dind" section.

https://gitlab.com/gitlab-org/gitlab-ce/issues/17861#note_12991518

Answer 3

I suppose there's no simple answer to your question. Before adding some details, I strongly suggest to read this blog article from the maintainer of DinD, which was originally named "do not use Docker in Docker for CI".

What you might try is declaring /var/lib/docker as a volume for your GitLab runner. But be warned, depending on your file-system drivers you may use AUFS in the container on an AUFS filesystem on your host, which is very likely to cause problems.

What I'd suggest to you is creating a separate Docker-VM, only for the runner(s), and bind-mount docker.sock from the VM into your runner-container. We are using this setup with GitLab with great success (>27.000 builds in about 12 months).

You can take a look at our runner with docker-compose support which is actually based on the shell-executor of GitLab's runner.