Can't mount file in gitlab CI with docker-compose

Question

We have a project that uses docker-compose for setup, so we'd like to use the same thing for tests (which we already have prepared and running locally), however I encountered 2 issues with gitlab's ci

ERROR: for store_k4g_nginx  Cannot start service nginx: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: rootfs_linux.go:59: mounting "/builds/project/whatever/docker/nginx/nginx.conf" to rootfs at "/var/lib/docker/overlay2/dfff5e2d38c01c8726cbd6fb46a44456521025d6cee3c596a6e56f7992938ce8/merged/etc/nginx/nginx.conf" caused: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

We're using our own runner with the following configuration

[[runners]]
  name = "******"
  url = "https://gitlab.com/"
  token = "*****"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "docker:stable"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
    shm_size = 0

and the ci script looks like this

test:
  image: docker/compose:latest
  stage: test
  variables:
    APP_ENV: test
  before_script:
    - apk add bash ncurses
  script:
    - bash ./scripts/test.sh --no-interaction --compose

Nginx dockerfile

FROM nginx:alpine
WORKDIR /var/www
RUN nginx -t;
CMD nginx
EXPOSE 80

And part of the docker-compose

  nginx:
    container_name: app_nginx
    build:
      context: ./nginx
    depends_on:
      - php-fpm
    volumes:
      - ../:/var/www
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf
      - ./nginx/sites:/etc/nginx/sites-available
      - ./nginx/conf.d:/etc/nginx/conf.d
      - ./logs/nginx:/var/log/nginx
    ports:
      - "${NGINX_PORT}:80"

the test.sh script brings up the project with docker-compose, which fails because of the error listed on top. Yes, nginx.conf is a file, it exists and works as expected in a local use of docker-compose. Another weird thing is that if I removed that line (which only happens for a nginx.conf file, linked to /etc/nginx/nginx.conf) and made it copy said file into the location in dockerfile it boots, but then it cannot see any files if i do ls -la

I believe we're running the gitlab runner natively and we've previously had nginx installed on the server. Could that be an issue? Should we run the runner in a docker as well?

Answer 1

Turns out that the gitlab runner was incorrectly configured. As we're running a docker-in-docker of sorts it's becoming rather annoying to set up the basic installation and stuff running, since it needs docker-compose and all.

The solution? Don't use DIND. It's vaguely explained on the gitlab documentation when they talk about it (I don't really get why - do people always only use 1 docker for their projects?), so once you have anything more than that and you don't actually copy your project files into the docker you're getting into a weird territory.

So the runner itself is running a docker, which then starts a docker (as seen in the CI gitlab file) which then spins up another docker.

I found an article that somewhat explains the issues related, but the quickest solution (and fastest for testing and build times, assuming you actually have your own runner) is that you mount the docker socket in the gitlab runner and then run the following configuration.

test:
  image: docker:latest
  stage: test
  variables:
    DOCKER_DRIVER: overlay2
    APP_ENV: test
  before_script:
    - apk add bash ncurses docker-compose
  script:
    - bash ./scripts/test.sh --no-interaction --compose

It builds fast (since the containers exist on the host machine, not docker-in-docker) and are cached (same reason).

As for why the file didn't mount - it's a bit more annoying. Since it was essentially 3 layers of dockers the first layer (host) has access to the files, so does second, but once the third one gets up - nope. That's why there were errors about the file not being there - it really was there, but a level higher.

After changing it to the configuration I've explained the application successfully builds without actually modifying the docker-compose or dockerfiles (since all files are properly visible from the second docker)