How to use docker-in-docker on locally installed gitlab-runner in privileged mode

Question

I have GitLab Runner installed on a spare computer that I'm using to run CI jobs by tagging certain jobs. However, some of the jobs in my pipeline require docker-in-docker I'm currently not able to run these on my local gitlab runner. Instead, I'm running them on the shared runners on gitlab.com. I have read through documentation that shows how to use privileged mode with the docker executor in gitlab-runner:

https://docs.gitlab.com/runner/executors/docker.html#use-docker-in-docker-with-privileged-mode

Here is the command I have used to register gitlab-runner on my machine:

sudo gitlab-runner register \
  --non-interactive \
  --url "https://gitlab.com/" \
  --registration-token "$PROJECT_REGISTRATION_TOKEN" \
  --executor "docker" \
  --docker-image alpine:latest \
  --description "docker-runner" \
  --tag-list "privileged" \
  --run-untagged="true" \
  --locked="false" \
  --access-level="not_protected"

Here is the /etc/gitlab-runner/config.toml that is generated by the above command:

sudo cat /etc/gitlab-runner/config.toml
[sudo] password for brian: 
concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "docker-runner"
  url = "https://gitlab.com/"
  token = "my-token-abcd-1234"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.docker]
    tls_verify = false
    image = "alpine:latest"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    pull_policy = "always"
    shm_size = 0
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

Here is the job that I'm trying to run in privileged mode:

Build backend:
  tags:
    - privileged
  stage: build
  image: docker:stable
  variables:
    DOCKER_HOST: tcp://docker:2375
    DOCKER_DRIVER: overlay2
  services:
    - docker:dind
  before_script:
    - |
      docker login \
        -u $CI_REGISTRY_USER \
        -p $CI_REGISTRY_PASSWORD \
        $CI_REGISTRY
  script:
    - |
      docker build \
        -t $CI_REGISTRY_IMAGE/backend:latest \
        -f backend/scripts/prod/Dockerfile .
    - docker push $CI_REGISTRY_IMAGE/backend:latest
  only:
    changes:
      - backend/**/*

Here is the error I'm seeing when I run this job:

$ docker login \ # collapsed multi-line command
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
$ docker build \ # collapsed multi-line command
Cannot connect to the Docker daemon at tcp://docker:2375. Is the docker daemon running?

I can see that the job is being picked up by my runner and that it is starting containers on my local machine, but it fails on the docker build command as shown above.

I found that following gitlab issue, and a tried a solution that seemed to work for lots of people: https://gitlab.com/gitlab-org/gitlab-runner/issues/1986#note_20339074

There seem to be lots of other solutions listed here and lots of confusion about the correct way to do this.

Answer 1

I solved the issue issue out as I was writing up the question, but I wanted to share to help make this solution easier to find.

The end of the above thread (https://gitlab.com/gitlab-org/gitlab-runner/issues/1986) links to a helpful article: https://about.gitlab.com/blog/2019/07/31/docker-in-docker-with-docker-19-dot-03/

Disabling TLS worked for me as described in the article.

Keeping everything the same as I described in my question, adding

DOCKER_TLS_CERTDIR: ""

to the variables section of the job allowed the job to proceed with no error.