Nginx Multi-Workload Config - Website & Reverse Proxy

Question

TLDR: I'm trying to configure Nginx to host a website AND act as a reverse proxy at the same time. The website is on 'www.example.com:443' and the reverse proxy is to a separate OpenVPN server on 'vpn.example.com:443'. I think I know the answer is "Nginx cannot be configured to listen on a single port for two different types of traffic" (web traffic and reverse proxy traffic), but thought I'd ask those more knowledgeable.

Now, on to the more long winded version. I do have a working setup that looks like this:

               +-----------------------+
               |                       |
               |         Nginx         |
               |    (Rev Proxy Only)   |
               |                       |
               |     192.168.0.100     |
               |  vpn.example.com:443  |
               |  www.example.com:443  |
               |                       |
               +-----------------------+
                 /                   \
                /                     \
+-----------------------+    +-----------------------+
|                       |    |                       |
|        OpenVPN        |    |       LAMP Stack      |
|         (VPN)         |    |        (Website)      |
|                       |    |                       |
|     192.168.0.101     |    |     192.168.0.102     |
|  vpn.example.com:443  |    |  www.example.com:443  |
|                       |    |                       |
+-----------------------+    +-----------------------+

Here are a couple of notes. The Nginx (v1.14.2) server is CentOS v7.6. It is configured with "--with-stream" and "--with-stream_ssl_preread_module". It also uses all defaults in the /etc/nginx/nginx.conf, except it includes a stream{} block with proxy_pass. The conf files look like this:

/etc/nginx/nginx.conf

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}

include /etc/nginx/proxy.conf;

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

/etc/nginx/proxy.conf

stream {

    map $ssl_preread_server_name $name {
        www.example.com lamp;
        vpn.example openvpn;
        default https_default_backend;
    }

    server {
        listen 192.168.0.100:443;
        proxy_pass $name;
        ssl_preread on;
    }

    upstream lamp {
        server 192.168.0.102:443;
    }

    upstream openvpn {
        server 192.168.0.101:443;
    }

    upstream https_default_backend {
        server 192.168.0.101:443;
    }

    log_format basic '[$time_local] $remote_addr - $protocol '
                     'STATUS: $status BYTES_SENT: $bytes_sent '
                     'BYTES_RECEIVED: $bytes_received '
                     'SENT_TO: $upstream_addr MSECS: $msec '
                     'SESSION_TIME: $session_time '
                     'UPSTREAM_BYTES_SENT: $upstream_bytes_sent '
                     'UPSTREAM_BYTES_RECEIVED: $upstream_bytes_received '
                     'UPSTREAM_CONNECTION_TIME: $upstream_connect_time"';

    access_log /var/log/nginx/stream.mfgs.dev_access.log basic;
    error_log  /var/log/nginx/stream.mfgs.dev_error.log;

}

Again, this is my current config, and everything here works. Website traffic is streamed to the web server and VPN traffic is streamed to the VPN server. All of this works.

My goal is to change the LAMP stack (Linux, Apache, MySQL, PHP) to a LEMP stack (Linux, Nginx, MySQL, PHP), and combine the Nginx reverse proxy and website onto the same server. Any traffic specifically directed to 'www.example.com:443' would go to the Nginx website and everything else would be passed over to the VPN server, like this:

+-----------------------+
|                       |
|         Nginx         |
| (Website & Rev Proxy) |
|                       |
|     192.168.0.100     |
|  vpn.example.com:443  |
|  www.example.com:443  |
|                       |
+-----------------------+
            |
            |
+-----------------------+
|                       |
|        OpenVPN        |
|         (VPN)         |
|                       |
|     192.168.0.101     |
|  vpn.example.com:443  |
|                       |
+-----------------------+

I've tried installing Nginx on a separate server (192.168.0.105). I got the website up and running on this server. However, when I include the proxy.conf, it tests correctly using 'nginx -t', but the Nginx service fails to restart on this server. The failure message is:

nginx: [emerg] bind() to 192.168.0.105:443 failed (98: Address already in use)

I believe that's because both the http and stream blocks are listening on the same IP & port. Is there some way to configure Nginx to act as a web server for a specific FQDN:port and proxy (via streaming) everything else to another server?

Thank you in advance.

Answer 1

I think you are trying to host www.example.com in same host as your new nginx host: 192.168.0.105.

If so, you will encounter address already in use error when start nginx service.

This is due to the both nginx stream and http service are configure to listen to port: 443:

http {
    server {
        listen 192.168.0.105:443;
        server_name www.example.com;
    }
}

stream {
    server {
        listen 192.168.0.105:443;
        proxy_pass $name;
        ssl_preread on;
    }
}

Unlike http service that allow named based server to work with same port. Nginx treat both http and stream as different service.

I have 2 solutions for far to solve the host stream and http in same node.

Solution 1: Let www.example.com listen to 127.0.0.1:443

As most nodes shall has a loopback interface by default. I believe you want to expose the stream:443 service directly instead of www.example.com:443. Let www.example.com listen to 127.0.0.1:443 isn't a bad idea to solve the address already in use issue.

http {
    server {
        listen 127.0.0.1:443;
        server_name www.example.com;
    }
}

Solution 2: Let www.example.com listen to a unix socket

Similar to solution 1 but using a unix socket for www.example.com to avoid additional TCP overhead for a little performance gain.

http {
    server {
        listen unix:/var/run/nginx.sock;
        server_name www.example.com;
    }
}

For both solutions, remember to re-configure the upstream in stream section with either 127.0.0.1 or unix socket.

I hope can find more alternative but this is what I can think of so far.