ipv4 https request not forwarded to docker container

Question

I have an annoying problem with Docker and my ovh server (ubuntu xenial). I run a NGINX official docker image (15.x) on my server and I forward some port from my server to my docker image (to host serveral services online).

All HTTP request done from outside (using IPV4 or IPV6) manage to reach NGINX. BUT, HTTPS requests done using IPV4 are not able to reach NGINX (IPV6 HTPPS works like a charm).

I listened to port 443 both on my server and in NGINX docker container with tcdump tcpdump -i any port 443 -s0 -n

Result: I saw IPV6 HTTPS request in both my server and NGINX container. I saw IPV4 https request only on my server but not in the NGINX container. => So it seems, this is not a "opened port" problem, because I managed to see IPV4 requests on my server (and I checked IPTables, 443 is forwarded to docker proxy). I think it's a port forwarding pb between my server and Docker proxy, only for HTTPS request done with IPV6.

Here is my container config:

nginx:
depends_on:
  - my_php7
image: nginx:latest
volumes:
  - "../:/usr/share/nginx/html:rw"
  - "./nginx/var/log:/var/log/nginx:rw"
  - "./nginx/nginx.template.conf:/etc/nginx/conf.d/nginx.template:rw"
  - "./nginx/sites-available:/etc/nginx/sites-available:rw"
  - "./nginx/sites-enabled:/etc/nginx/sites-enabled:rw"
links:
  - my_php7:webstack_php
ports:
  - "8080:8080"
  - "8090:90"
  - "8091:91"
  - "8083:8083"
  - "80:80"
  - "443:443"
restart: always
command: /bin/bash -c "cp /etc/nginx/conf.d/nginx.template /etc/nginx/conf.d/nginx.conf && nginx -c /etc/nginx/conf.d/nginx.conf"

I checked also net.ipv6.bindv6only:
#> sysctl net.ipv6.bindv6only net.ipv6.bindv6only = 0

here are my nat infos:

Proto Recv-Q Send-Q Adresse locale          Adresse distante        Etat       PID/Program name
tcp        0      0 0.0.0.0:8081            0.0.0.0:*               LISTEN      19671/node
tcp        0      0 91.121.101.165:914      0.0.0.0:*               LISTEN      1696/openvpn-openss
tcp        0      0 91.121.101.165:915      0.0.0.0:*               LISTEN      1724/openvpn-openss
tcp        0      0 0.0.0.0:8084            0.0.0.0:*               LISTEN      22633/node
tcp        0      0 91.121.101.165:916      0.0.0.0:*               LISTEN      1751/openvpn-openss
tcp        0      0 91.121.101.165:917      0.0.0.0:*               LISTEN      1779/openvpn-openss
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      929/named
tcp        0      0 0.0.0.0:21              0.0.0.0:*               LISTEN      947/vsftpd
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      943/sshd
tcp        0      0 127.0.0.1:953           0.0.0.0:*               LISTEN      929/named
tcp        0      0 127.0.0.1:4040          0.0.0.0:*               LISTEN      4765/ngrok
tcp6       0      0 :::8082                 :::*                    LISTEN      12095/docker-proxy
tcp6       0      0 :::8083                 :::*                    LISTEN      12837/docker-proxy
tcp6       0      0 :::2003                 :::*                    LISTEN      12058/docker-proxy
tcp6       0      0 :::9300                 :::*                    LISTEN      12006/docker-proxy
tcp6       0      0 :::85                   :::*                    LISTEN      12070/docker-proxy
tcp6       0      0 ::1:53                  :::*                    LISTEN      929/named
tcp6       0      0 :::22                   :::*                    LISTEN      943/sshd
tcp6       0      0 :::3000                 :::*                    LISTEN      12544/docker-proxy
tcp6       0      0 ::1:953                 :::*                    LISTEN      929/named
tcp6       0      0 :::8090                 :::*                    LISTEN      12861/docker-proxy
tcp6       0      0 :::443                  :::*                    LISTEN      12885/docker-proxy
tcp6       0      0 :::8091                 :::*                    LISTEN      12849/docker-proxy
tcp6       0      0 :::92                   :::*                    LISTEN      12082/docker-proxy
tcp6       0      0 :::4000                 :::*                    LISTEN      12532/docker-proxy
tcp6       0      0 :::9000                 :::*                    LISTEN      12521/docker-proxy
tcp6       0      0 :::5000                 :::*                    LISTEN      12106/docker-proxy
tcp6       0      0 :::3306                 :::*                    LISTEN      12119/docker-proxy
tcp6       0      0 :::80                   :::*                    LISTEN      12897/docker-proxy
tcp6       0      0 :::8080                 :::*                    LISTEN      12873/docker-proxy
tcp6       0      0 :::9200                 :::*                    LISTEN      12035/docker-proxy
tcp6       0      0 :::4400                 :::*                    LISTEN      28319/node
udp6    3072      0 :::40195                :::*                                1081/collectd
udp6       0      0 :::8124                 :::*                                12046/docker-proxy
udp6       0      0 ::1:53                  :::*                                929/named
udp6       0      0 ff18::efc0:4a42:25826   :::*                                1081/collectd

For those who wonder why there is only IPV6 forwarding displayed for 443:

there is the same for 80 port and it s working fine
From https://groups.google.com/d/msg/golang-nuts/F5HE7Eqb6iM/q_um2VqT5vAJ

on linux, by default, net.ipv6.bindv6only is 0, so ipv4 packets could also be received from ipv6 sockets with ipv4-mapped ipv6 address. thus you only need to listen on tcp6 socket and we can support both ipv4 and ipv6.

if you want explicitly only listen on ipv4 port, you will have to use net.Listen("tcp4", "0.0.0.0:3000") and then pass the listener to http.Serve.

I tried to rebuild the container without success. I also tried to open 443 port in another docker => it was not able to get IPV4 HTTPS requests.

To resume:

All HTTP requests (IPV4/IPV6) works
Only IPV6 HTTPS works, IPV4 seems to not beeing forwarded to Docker
IPtables, Docker conf and net stats are ok.

any ideas? :)

[edit]

@leodotcloud

sure here it is (I hide the server name with xxxx)

The vhost config

server {
    listen 443 ssl http2;

    server_name www.xxx.com;

    error_log /var/log/nginx/error.log;
    access_log /var/log/nginx/access.log;

   ssl_certificate    /etc/letsencrypt/live/xxx.com/fullchain.pem;
   ssl_certificate_key  /etc/letsencrypt/live/xxxx.com/privkey.pem;
   ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
   ssl_session_timeout  10m;
   ssl_session_cache shared:SSL:10m;
   ssl_prefer_server_ciphers on;

    charset utf-8;

    location / {
        proxy_pass http://node;
        proxy_set_header X-Real-IP  $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
    }

}

The nginx conf for SSL

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    ##
    # SSL Settings
    ##

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

....

}

[edit 2] I tried to change my docker-compose configuration from "443:443" to "0.0.0.0:443:443" to force ipv4 binding .

And I got this error : Bind for 0.0.0.0:443 failed: port is already allocated

So I have something which use my IPV4 connexion on 443.

When I use the "443:443" syntax to bind IPV6 and IPV4, docker bind IPV6 but don't check IPV4 (maybe it's the system who care about binding IPV4 too).

Now I have to find this service. Problem : I found it nowhere using netstat. Is there a way to force closing a socket?

[EDIT] I try stopping containers and start nginx directly on the server, bind on 443 +SSL IPV4/IPV6 and serve a "hello" file . => IPV6 works, IPV4 don't work => I change the port from 443 to 444 => IPV4 and IPV6 works => I clear IPTABLE and configure ACCEPT on everything => Still same problem with IPV4 443 :(

Because Nginx manage to bind on 0.0.0.0:443 I suppose the socket was "free". So something is blocking or filtering data incoming from IPV4 443

[edit] Finally it was an IPTABLE issue. I tag my answer as the solution to re-open port 443 in IPV4. I will look in my NAT table to find which rule is causing this and I will open a new topic if needed . tahnks everyone for yout ime and help !

wulfy · Accepted Answer

I finally found a way to make it work. Just FLUSHED nat using the below command:

iptables -t nat -F

And now everything works. :)

I don't really understand what this command does. It flushes the NAT rules but I can't see any difference in my IPTABLE.

If someone can tell me if this "flush" is risky (I never changed something in IPTABLE and only docker change it).

ipv4 https request not forwarded to docker container

Answers (2)

Related Questions