Jenkins in docker with access to host docker

Question

I have a workflow as follows for publishing webapps to my dev server. The server has a single docker host and I'm using docker-compose for managing containers.

1. Push changes in my app to a private gitlab (running in docker). The app includes a Dockerfile and docker-compose.yml
2. Gitlab triggers a jenkins build (jenkins is also running in docker), which does some normal build stuff (e.g. run test)
3. Jenkins then needs to build a new docker image and deploy it using docker-compose.

The problem I have is in step 3. The way I have it set up, the jenkins container has access to the host docker so that running any docker command in the build script is essentially the same as running it on the host. This is done using the following DockerFile for jenkins:

FROM jenkins
USER root

# Give jenkins access to docker
RUN groupadd -g 997 docker
RUN gpasswd -a jenkins docker

# Install docker-compose
RUN curl -L https://github.com/docker/compose/releases/download/1.2.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
RUN chmod +x /usr/local/bin/docker-compose

USER jenkins

and mapping the following volumes to the jenkins container:

-v /var/run/docker.sock:/var/run/docker.sock
-v /usr/bin/docker:/usr/bin/docker

A typical build script in jenkins looks something like this:

docker-compose build
docker-compose up

This works ok, but there are two problems:

• It really feels like a hack. But the only other options I've found is to use the docker plugin for jenkins, publish to a registry and then have some way of letting the host know it needs to restart. This is quite a lot more moving parts, and the docker-jenkins plugin required that the docker host is on an open port, which I don't really want to expose.

• The jenkins DockerFile includes groupadd -g 997 docker which is needed to give the jenkins user access to docker. However, the GID (997) is the GID on the host machine, and is therefore not portable.

I'm not really sure what solution I'm looking for. I can't see any practical way to get around this approach, but it would be nice if there was a way to allow running docker commands inside the jenkins container without having to hard code the GID in the DockerFile. Does anyone have any suggestions about this?

Answer 1

I solved a similar problem in the following way. Docker is installed on the host. Jenkins is deployed in the docker container of the host. Jenkins must build and run containers with web applications on the host.

Jenkins master connects to the docker host using REST APIs. So we need to enable the remote API for our docker host.

Log in to the host and open the docker service file /lib/systemd/system/docker.service. Search for ExecStart and replace that line with the following.

ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:4243 -H unix:///var/run/docker.sock

Reload and restart docker service

sudo systemctl daemon-reload
sudo service docker restart

Docker file for Jenkins

FROM jenkins/jenkins:lts
USER root

# Install the latest Docker CE binaries and add user `jenkins` to the docker group
RUN apt-get update
RUN apt-get -y --no-install-recommends install apt-transport-https \
      apt-utils ca-certificates curl gnupg2 software-properties-common && \
    curl -fsSL https://download.docker.com/linux/$(. /etc/os-release; echo     "$ID")/gpg > /tmp/dkey; apt-key add /tmp/dkey && \
    add-apt-repository \
      "deb [arch=amd64] https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") \
      $(lsb_release -cs) \
      stable"

RUN apt-get update && apt-get install -y docker-ce-cli docker-ce && \
   apt-get clean && \
   usermod -aG docker jenkins

USER jenkins
RUN jenkins-plugin-cli --plugins "blueocean:1.25.6 docker-workflow:1.29 ansicolor"

Build jenkins docker image

  docker build -t you-jenkins-name .

Run Jenkins

  docker run --name you-jenkins-name --restart=on-failure --detach \
  --network jenkins \
  --env DOCKER_HOST=tcp://172.17.0.1:4243 \
  --publish 8080:8080 --publish 50000:50000 \
  --volume jenkins-data:/var/jenkins_home \
  --volume jenkins-docker-certs:/certs/client:ro \
  you-jenkins-name  

Your web application has a repository at the root of which is jenkins and a docker file. Jenkinsfile for web app:

    pipeline {
        agent any
    
        environment {
            PRODUCT = 'web-app'
            HTTP_PORT = 8082
            DEVICE_CONF_HOST_PATH = '/var/web-app'
        }
    
        options {
            ansiColor('xterm')
            skipDefaultCheckout()
        }
    
        stages {
            stage('Checkout') {
                steps {
                    script {
                        //BRANCH_NAME = env.CHANGE_BRANCH ? env.CHANGE_BRANCH : env.BRANCH_NAME
                        deleteDir()
                        //git url: "git@<host>:<org>/${env.PRODUCT}.git", branch: BRANCH_NAME
                    }
                    checkout scm
                }
            }
            stage('Stop and remove old') {
                steps {
                    script {
                        try {
                            sh "docker stop ${env.PRODUCT}"
                        } catch (Exception e) {}
                        try {
                            sh "docker rm ${env.PRODUCT}"
                        } catch (Exception e) {}
                        try {
                            sh "docker image rm ${env.PRODUCT}"
                        } catch (Exception e) {}
                    }
                }
            }
    
            stage('Build') {
                steps {
                    sh "docker build . -t ${env.PRODUCT}"
                }
            }
    
            // ④ Run the test using the built docker image
            stage('Run new') {
                steps {
                    script {
                        sh """docker run
                         --detach
                         --name ${env.PRODUCT} \
                        --publish ${env.HTTP_PORT}:8080 \
                        --volume ${env.DEVICE_CONF_HOST_PATH}:/var/web-app \
                        ${env.PRODUCT} """
                    }
                }
            }
        }
    }

Answer 2

I ran into the same issues. I ended up giving Jenkins passwordless sudo privileges because of the GID problem. I wrote more about this here: https://blog.container-solutions.com/running-docker-in-jenkins-in-docker

This doesn't really affect security as having docker privileges is effectively equivalent to sudo rights.

Answer 3

My previous answer was more generic, telling how you can modify the GID inside the container at runtime. Now, by coincidence, someone from my close colleagues asked for a jenkins instance that can do docker development so I created this:

FROM bdruemen/jenkins-uid-from-volume
RUN apt-get -yqq update && apt-get -yqq install docker.io && usermod -g docker jenkins
VOLUME /var/run/docker.sock
ENTRYPOINT groupmod -g $(stat -c "%g" /var/run/docker.sock) docker && usermod -u $(stat -c "%u" /var/jenkins_home) jenkins && gosu jenkins /bin/tini -- /usr/local/bin/jenkins.sh

(The parent Dockerfile is the same one I have described in my answer to: Changing the user's uid in a pre-build docker container (jenkins))

To use it, mount both, jenkins_home and docker.sock.

docker run -d /home/jenkins:/var/jenkins_home -v /var/run/docker.sock:/var/run/docker.sock <IMAGE>

The jenkins process in the container will have the same UID as the mounted host directory. Assuming the docker socket is accessible to the docker group on the host, there is a group created in the container, also named docker, with the same GID.

Answer 4

Please take a look at this docker file I just posted: https://github.com/bdruemen/jenkins-docker-uid-from-volume/blob/master/gid-from-volume/Dockerfile

Here the GID extracted from a mounted volume (host directory), with

stat -c '%g' <VOLUME-PATH>

Then the GID of the group of the container user is changed to the same value with

groupmod -g <GID>

This has to be done as root, but then root privileges are dropped with

gosu <USERNAME> <COMMAND>

Everything is done in the ENTRYPOINT, so the real GID is unknown until you run

docker run -d -v <HOST-DIRECTORY>:<VOLUME-PATH> ...

Note that after changing the GID, there might be other files in the container no longer accessible for the process, so you might need a

chgrp -R <GROUPNAME> <SOME-PATH>    

before the gosu command.

You can also change the UID, see my answer here Changing the user's uid in a pre-build docker container (jenkins) and maybe you want to change both to increase security.