CODEWITHSUNDEEP
dockergitlabdocker-composegitlab-ci
Craig Otis
Craig Otis

Reputation: 32104

Gitlab CI runner not able to expose ports of nested Docker containers

When using GitLab CI, as well as the gitlab-ci-multi-runner, I'm unable to get internally-started Docker containers to expose their ports to the "host", which is the Docker image in which the build is running.

My .gitlab-ci.yml file:

test:
  image: docker
  stage: test
  services:
    - docker:dind
  script:
    - APP_CONTAINER_ID=`docker run -d --privileged -p "9143:9143" appropriate/nc nc -l 9143`
    - netstat -a
    - docker exec $APP_CONTAINER_ID netstat -a
    - nc -v localhost 9143

My command:

gitlab-ci-multi-runner exec docker --docker-privileged test

The output:

$ netstat -a
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 runner--project-1-concurrent-0:54664 docker:2375             TIME_WAIT
tcp        0      0 runner--project-1-concurrent-0:54666 docker:2375             TIME_WAIT
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node Path

$ docker exec $APP_CONTAINER_ID netstat -a
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:9143            0.0.0.0:*               LISTEN
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node Path

$ nc -v localhost 9143
ERROR: Build failed: exit code 1
FATAL: exit code 1

What am I doing wrong here?

Original Question Follows - above is a shorter, easier-to-test example

I have an application image that listens on port 9143. Its startup and config is managed via docker-compose.yml, and works great on my local machine with docker-compose up - I can access localhost:9143 without issue.

However, when running on GitLab CI (the gitlab.com version) via a shared runner, the port doesn't seem to be exposed.

The relevant portion of my .gitlab-ci.yml:

test:
  image: craigotis/buildtools:v1
  stage: test
  script:
    - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN registry.gitlab.com/craigotis/myapp
    - docker-compose up -d
    - sleep 60 # a temporary hack to get the logs
    - docker-compose logs
    - docker-machine env
    - docker-compose port app 9143
    - netstat -a
    - docker-compose ps
    - /usr/local/bin/wait-for-it.sh -h localhost -p 9143 -t 60
    - cd mocha
    - npm i
    - npm test
    - docker-compose down

The output is:

$ docker-compose logs
...
app_1  | [Thread-1] INFO spark.webserver.SparkServer - == Spark has ignited ...
app_1  | [Thread-1] INFO spark.webserver.SparkServer - >> Listening on 0.0.0.0:9143
app_1  | [Thread-1] INFO org.eclipse.jetty.server.Server - jetty-9.0.z-SNAPSHOT
app_1  | [Thread-1] INFO org.eclipse.jetty.server.ServerConnector - Started ServerConnector@6919dc5{HTTP/1.1}{0.0.0.0:9143}
...

$ docker-compose port app 9143
0.0.0.0:9143

$ netstat -a
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       
tcp        0      0 runner-e11ae361-project-1925166-concurrent-0:53646 docker:2375             TIME_WAIT   
tcp        0      0 runner-e11ae361-project-1925166-concurrent-0:53644 docker:2375             TIME_WAIT   
tcp        0      0 runner-e11ae361-project-1925166-concurrent-0:53642 docker:2375             TIME_WAIT   
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node Path

$ docker-compose ps
stty: standard input: Not a tty
    Name                  Command               State                Ports               
----------------------------------------------------------------------------------------
my_app_1   wait-for-it.sh mysql_serve ...   Up      8080/tcp, 0.0.0.0:9143->9143/tcp 
mysql_server   docker-entrypoint.sh --cha ...   Up      3306/tcp     

$ /usr/local/bin/wait-for-it.sh -h localhost -p 9143 -t 60
wait-for-it.sh: waiting 60 seconds for localhost:9143
wait-for-it.sh: timeout occurred after waiting 60 seconds for localhost:9143

The contents of my docker-compose.yml:

version: '2'

networks:
    app_net:
        driver: bridge

services:
    app:
        image: registry.gitlab.com/craigotis/myapp:latest
        depends_on:
        - "db"
        networks:
        - app_net
        command: wait-for-it.sh mysql_server:3306 -t 60 -- java -jar /opt/app*.jar
        ports:
        - "9143:9143"

    db:
        image: mysql:latest
        networks:
        - app_net
        container_name: mysql_server
        environment:
        - MYSQL_ALLOW_EMPTY_PASSWORD=true

It seems like my application container is listening on 9143, and it's properly exposed to the shared GitLab runner, but it doesn't seem to actually be exposed. It works fine on my local machine - is there some special workaround/tweak I need to make this work inside a Docker container running on GitLab?

Upvotes: 29

Views: 20931

Answers (5)

MingalevME
MingalevME

Reputation: 1985

In case you GitLab CI Runner run with Docker-executor via socket binding use host.docker.internal-host, because your app is running on the host machine but not on the guest.

# Run and ssh into dind-container
docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock docker:20.10-dind /bin/sh

# Next commands run in dind-container

docker run --rm -d -p 8888:80 nginx:alpine
...

wget -qO- "http://localhost:8888/"
wget: can't connect to remote host (127.0.0.1): Connection refused

wget -qO- "http://127.0.0.1:8888/"
wget: can't connect to remote host (127.0.0.1): Connection refused

wget -qO- "http://docker:8888/"
wget: bad address 'docker:8888'

wget -qO- "http://host.docker.internal:8888/"
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

Upvotes: 2

Jason Prawn
Jason Prawn

Reputation: 1053

When using docker:dind a container is created and your docker-compose containers get setup within it. It exposes the ports to localhost within the docker:dind container. You cannot access this as localhost from the environment that your code is executing in.

A hostname of docker is setup for you to reference this docker:dind container. You can check by using cat /etc/hosts.

Instead of referencing localhost:9143 you should use docker:9143.

Upvotes: 21

Rohit Dhiman
Rohit Dhiman

Reputation: 2741

Your docker-compose.yml seems to be ok.

But I think there is error with your ip or port routing. As I can see from your shared info your app is running on port 9143 on ip 0.0.0.0 as 0.0.0.0:9143.

and you are accessing it as localhost:9143, which can be interpreted as 127.0.0.1:9143.

According to this.

127.0.0.1 is the loopback address (also known as localhost).
0.0.0.0 is a non-routable meta-address used to designate an invalid, unknown, or non-applicable target (a ‘no particular address’ place holder).

Can you try to run your app on 127.0.0.1:9143 then share the result.

UPDATE

or you can use service to run it by service name as documentation suggest:

The services keyword defines just another docker image that is run during your build and is linked to the docker image that the image keyword defines. This allows you to access the service image during build time.

The service container for MySQL will be accessible under the hostname mysql. So, in order to access your database service you have to connect to the host named mysql instead of a socket or localhost.

Upvotes: 1

Yoav Aharoni
Yoav Aharoni

Reputation: 2672

Usually a docker machine won't run on localhost, but on a docker host with some other ip address. Try using docker-machine ip to get your docker host ip.

Upvotes: 0

VonC
VonC

Reputation: 1329782

The offical gitab-ci on gitlab.com documentation refers to the example of PostgreSQL

Its working CI does not try to connect to localhost, but rather to the service name

The services keyword defines just another docker image that is run during your build and is linked to the docker image that the image keyword defines. This allows you to access the service image during build time.

The service container for MySQL will be accessible under the hostname mysql.
So, in order to access your database service you have to connect to the host named mysql instead of a socket or localhost.

You could check if this applies in your case, and try accessing your application service in app:9143 instead of localhost:9143.

Upvotes: 13

Related Questions