Docker container cannot access internet, only ping works

Question

After hours of searching and trying solutions I am at an end.

When I start a Docker container on a Centos7 machine, this is what happens:

• Cannot communicate via TCP (e.g. run apt update). ping works fine.
• Switching network mode to --net=host fixes the issue, but I want to use the default bridge
• I added net.ipv4.ip_forward=1 to /etc/sysctl.conf
• Container is registered in the bridge network seen via docker network inspect bridge
• DNS is configured properly, pinging hostnames works as well as addresses.

Further info

Running apt update shows messages such as:

Ign http://security.debian.org jessie/updates InRelease
Err http://security.debian.org jessie/updates Release.gpg
  Unable to connect to security.debian.org:http: [IP: 212.211.132.32 80]

iptables configuration:

$ iptables -L
Chain INPUT (policy DROP)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     icmp --  anywhere             anywhere
           tcp  --  anywhere             anywhere             tcp dpt:http /* HTTP-IN */
           tcp  --  anywhere             anywhere             tcp dpt:http flags:FIN,SYN,RST,ACK/SYN /* HTTP-SYN */
           tcp  --  anywhere             anywhere             tcp dpt:https /* HTTPS-IN */
           tcp  --  anywhere             anywhere             tcp dpt:https flags:FIN,SYN,RST,ACK/SYN /* HTTPS-SYN */
ACCEPT     tcp  --  anywhere             anywhere             tcp flags:!FIN,SYN,RST,ACK/SYN
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:https
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh match-set avast_internal src
ACCEPT     udp  --  anywhere             anywhere             udp dpt:snmp match-set avast_internal src
ACCEPT     tcp  --  anywhere             anywhere             tcp match-set avast_internal src

Chain FORWARD (policy DROP)
target     prot opt source               destination
DOCKER-ISOLATION  all  --  anywhere             anywhere
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
           tcp  --  anywhere             anywhere             tcp spt:http /* HTTP-OUT */
           tcp  --  anywhere             anywhere             tcp spt:https /* HTTPS-OUT */

Chain DOCKER (2 references)
target     prot opt source               destination

Chain DOCKER-ISOLATION (1 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere
DROP       all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

What could be the cause of this?

---

Edit:

After opening the firewall using these commands I was able to connect to the internet from the docker container:

sudo iptables -P INPUT ACCEPT
sudo iptables -P FORWARD ACCEPT
sudo iptables -P OUTPUT ACCEPT

sudo iptables -t nat -F
sudo iptables -t mangle -F
sudo iptables -F
sudo iptables -X

However, after resetting the rules to find out which exactly caused the problem, I am unable to get it back to a working state, even after clearing all iptables rules and restarting docker daemon.

---

Edit 2:

Turns out there were some rules in the raw table as well, which I did not see before. The following rule was blocking Docker containers from the internet:

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination

CT         tcp  --  anywhere             anywhere             NOTRACK

Answer 1

As you said

Ign http://security.debian.org jessie/updates InRelease
Err http://security.debian.org jessie/updates Release.gpg
  Unable to connect to security.debian.org:http: [IP: 212.211.132.32 80]

This kind of problem arises when the DNS is not confgured for the Docker Containers.[similar problem in my environment as well].

The following procedure will help in resolving the DNS issue while working with DOCKER containers.

First step is to check the external connectivity from the containers:

# docker run busybox ping -c 2 192.203.230.10
PING 192.203.230.10 (192.203.230.10): 56 data bytes
64 bytes from 192.203.230.10: seq=0 ttl=56 time=66.724 ms
64 bytes from 192.203.230.10: seq=1 ttl=56 time=54.786 ms

When you try to ping to google.com using the container it fails to reach because of DNS issue.

# docker run busybox nslookup google.com
Server:    8.8.8.8
Address 1: 8.8.8.8
nslookup: can't resolve 'google.com'

Find out the DNS server used in your machine :

# nm-tool  |grep DNS
    DNS:             172.24.100.50
    DNS:             10.1.100.50

Run it again using DNS IP found in the above step which resolves the DNS issue:

# docker run --dns 172.24.100.50 busybox nslookup google.com
Server:    172.24.100.50
Address 1: 172.24.100.50 indc01.radisys.com
Name:      google.com
Address 1: 2607:f8b0:4009:80c::200e ord36s01-in-x0e.1e100.net
Address 2: 172.217.4.110 ord36s04-in-f14.1e100.net

To resolve it permanently add the following content as below to a new file:

# cat /etc/docker/daemon.json
{
    "dns" : ["172.24.100.50", "8.8.8.8"]
}

More info on Docker DNS configuration : https://docs.docker.com/engine/userguide/networking/configure-dns/

Restart the docker service and check the connectivity again:

# docker run busybox nslookup google.com
Server:    172.24.100.50
Address 1: 172.24.100.50 indc01.radisys.com
Name:      google.com
Address 1: 2607:f8b0:4009:801::200e ord30s31-in-x0e.1e100.net
Address 2: 172.217.4.238 ord30s31-in-f14.1e100.net

Check it by running another container:

# docker run -it e02e811dd08f
/ # ping google.com
PING google.com (172.217.4.238): 56 data bytes
64 bytes from 172.217.4.238: seq=0 ttl=47 time=251.506 ms
64 bytes from 172.217.4.238: seq=1 ttl=47 time=245.621 ms

Answer 2

Turns out there were some rules in the raw table as well, which I did not see before. The following rule was blocking Docker containers from the internet:

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination

CT         tcp  --  anywhere             anywhere             NOTRACK

Deleting this rule fixed the problem.