Docker-Compose: connect to host when host.docker.internal doesn't work

Question

My eventual goal here is to allow a container running a FastAPI app to communicate with a MySQL database on the host.

First I tried using host.docker.internal

Dockerfile

FROM debian:latest

RUN apt update && apt install -y \
    netcat \
    iputils-ping

CMD echo "tailing /dev/null" && tail -f /dev/null

docker-compose.yml

version: "3.2"

services:
  test:
    build:
      context: "."
    extra_hosts:
      - "host.docker.internal:host-gateway"

Expected behavior: ping works, nc -vz works

In particular, with nc -vz I'd expect to see something like:

root@9fe8de220d44:/# nc -vz host.docker.internal 80
Connection to host.docker.internal (172.17.0.1) port 80 (tcp) succeeded!

Actual behavior: ping works, nc -vz doesn't

root@5981bcfbf598:/# ping host.docker.internal
PING host.docker.internal (172.17.0.1) 56(84) bytes of data.
64 bytes from host.docker.internal (172.17.0.1): icmp_seq=1 ttl=64 time=0.079 ms
64 bytes from host.docker.internal (172.17.0.1): icmp_seq=2 ttl=64 time=0.067 ms
64 bytes from host.docker.internal (172.17.0.1): icmp_seq=3 ttl=64 time=0.068 ms
^C
--- host.docker.internal ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2013ms
rtt min/avg/max/mdev = 0.067/0.071/0.079/0.005 ms
root@5981bcfbf598:/# nc -vz host.docker.internal 80
nc: connect to host.docker.internal (172.17.0.1) port 80 (tcp) failed: Connection refused

On the host

I have apache running on port 80

$ netstat -tulpn
...
tcp6       0      0 :::80                   :::*                    LISTEN      1258/apache2

Additionally, my firewall is configured to allow all inbound requests to port 80: firewall says http port 80 allows all ipv4 and ipv6

OS and docker versions:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.6 LTS
Release:    18.04
Codename:   bionic
$ docker --version
Docker version 20.10.21, build baeda1f

Manually specifying the network also fails

After host.docker.internal failed, I followed the instructions for connecting from a container to a Linux host (Ubuntu 18.04 in my case) here using a manually specified network: https://stackoverflow.com/a/70725882

Here's my setup:

Dockerfile

FROM debian:latest

RUN apt update && apt install -y \
    netcat \
    iputils-ping

CMD echo "tailing /dev/null" && tail -f /dev/null

docker-compose.yml

version: "3.2"

networks:
  test:
    name: test-network
    attachable: true
    ipam:
      driver: default
      config:
        - subnet: 172.42.0.0/16
          ip_range: 172.42.5.0/24
          gateway: 172.42.0.1

services:
  test:
    build:
      context: "."
    networks:
      - test

Confirm gateway

$ docker inspect test-test-1  -f '{{range .NetworkSettings.Networks}}{{.Gateway}}{{end}}'
172.42.0.1

ping works

root@07f81c211a0c:/# ping 172.42.0.1
PING 172.42.0.1 (172.42.0.1) 56(84) bytes of data.
64 bytes from 172.42.0.1: icmp_seq=1 ttl=64 time=0.056 ms
64 bytes from 172.42.0.1: icmp_seq=2 ttl=64 time=0.068 ms
64 bytes from 172.42.0.1: icmp_seq=3 ttl=64 time=0.065 ms

Expected behavior: nc -vz succeeds

From the instructions at https://stackoverflow.com/a/70725882:

root@9fe8de220d44:/# nc -vz 172.18.0.1 80
Connection to 172.18.0.1 80 port [tcp/http] succeeded!

Actual behavior: nc -vz fails

root@07f81c211a0c:/# nc -vz 172.42.0.1 80
nc: connect to 172.42.0.1 port 80 (tcp) failed: Connection refused

What am I doing wrong?

Thanks in advance for your help!

Answer 1

Update

The answer below works for the general problem of connecting to the host, but there's a much simpler solution if you're trying to expose a particular service that has a socket: mount the socket! For example, if you want to connect to a local mysql, then in your docker-compose.yml you can simply add:

    volumes:
      - /var/run/mysqld/mysqld.sock:/var/run/mysqld/mysqld.sock:ro

to whichever service needs to communicate with the host mysql. Easy peasy. If you're using the url syntax for mysql, you then specify that you want to use the unix socket, eg, for sqlalchemy:

mysql+pymysql://user:passwd@host/db?unix_socket=/var/run/mysqld/mysqld.sock

Original answer

I was able to solve this problem thanks to https://forums.docker.com/t/how-to-connect-from-docker-container-to-the-host/123318.

I ran

$ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet XXX.XXX.XXX.XXX  netmask ...

and copied the address XXX.XXX.XXX.XXX, and replaced host-gateway with it in my docker-compose.yml:

version: "3.2"

services:
  test:
    build:
      context: "."
    extra_hosts:
      - "host.docker.internal:XXX.XXX.XXX.XXX"

Now, from within the container:

root@f5836a37815a:/# nc -vz host.docker.internal 80
Connection to host.docker.internal (104.248.221.215) 80 port [tcp/*] succeeded!

I'm not sure why all the advice to use host-gateway didn't work. I was under the impression it should work for my version of docker (compose).

$ docker version
Client: Docker Engine - Community
 Version:           20.10.21
...
$ docker compose version
Docker Compose version v2.12.2