haproxy 1.5 specific source IP address show <NOSRV> 503 SC in haproxy log

Question

I have an unusual problem (as everyone does on this site). I have HAProxy listening on port 80 and 443. I am offloading SSL encryption to HAproxy and passing all traffic to my web servers on port 80. There is a firewall in front of my HAProxy server with NAT rules pointing to my internal listening IP addresses of HAProxy. We are getting an unusual 503 message for random IP source addresses over SSL on port 443. Other random IP addresses are working fine over SSL. Our service is an API and most traffic goes through but a very small percentage gets a 503.

Here is an HAProxy log entry that works and one that doesn't working:

localhost haproxy[5404]: XXX.XXX.XXX.XXX:54787 [15/Jun/2016:22:46:57.592] https_in_ssl~ http_www2/web1 32/0/0/232/264 200 747 - - ---- 5/4/0/1/0 0/0 "POST /webservices/ourService.asmx HTTP/1.1"

not working:

localhost haproxy[5404]: XXX.XXX.XXX.XXX:55494 [15/Jun/2016:22:46:39.514] https_in_ssl~ https_in_ssl/<NOSRV> -1/-1/-1/-1/227 503 212 - - SC-- 3/2/0/0/0 0/0 "POST /webservices/ourService.asmx HTTP/1.0"

One thing I notice is the front-end and back-end are the same in the not working log entry.

Here is my config file:

peers prodHAproxypeers
        peer haproxylb1 10.0.0.145:1024
        peer haproxylb2 10.0.0.146:1024

global
        log     127.0.0.1 local0
#       log /dev/log local0
#       log /dev/log local1 notice
        chroot /var/lib/haproxy
        stats socket /var/lib/haproxy/stats
        stats timeout 30s
        tune.ssl.default-dh-param 2048
        user haproxy
        group haproxy
        daemon

defaults
        log global
        mode http
        option httplog
        option dontlognull
        option redispatch
        option forwardfor
        option http-server-close
        maxconn 5000
        timeout connect 5s
        timeout client 5h
        timeout server 5h
        timeout queue 30s
        timeout http-request 5s
        timeout http-keep-alive 15s

listen stats *:1936
        mode http
        stats enable
        stats hide-version
        stats realm Haproxy\ Statistics
        stats uri /haproxy_stats
        stats auth admin:hardPassword
        stats admin if TRUE

frontend http_in
        bind *:80
        ###Add new acl and use_backend entry for each new site
        ###new backend sections will be needed as well
        acl is_www1 hdr(host) -i www1.domainname.com
        acl is_www2 hdr(host) -i www2.domainname.com
        acl is_www3 hdr(host) -i www3.domainname.com
        acl is_www4 hdr(host) -i www4.domainname.com
        acl is_wildcardwww hdr_end(host) -i domainname.com
        use_backend http_www1 if is_www1
        use_backend http_www2 if is_www2
        use_backend http_www3 if is_www3
        use_backend http_www4 if is_www4
        use_backend http_www5 if is_www5
        option forwardfor
        option http-server-close

frontend https_in_ssl
        mode http
        bind *:443 ssl crt /etc/ssl/private/ no-sslv3
        reqadd X-Forwarded-Proto:\ https
        use_backend http_www1 if { ssl_fc_sni www1.domainname.com }
        use_backend http_www2 if { ssl_fc_sni www2.domainname.com }

        acl is_ssl_www5 hdr_end(host) -i domainname.com
        use_backend http_www5 if is_ssl_www5


backend http_www1
        balance source
        cookie SRV_ID prefix
        stick-table type ip size 1m expire 6h peers prodHAproxypeers
        stick on src
        ###This site does not use host header - only the page name is needed###
#       option httpchk HEAD /Default.aspx
        ###Added host header so haproxy can route around NLB - use below for checking###
        option httpchk HEAD /Default.aspx HTTP/1.1\r\nHost:\ www1.domainname.com
        server p-websvr01 10.0.0.10:80 cookie pweb1 weight 45 check
        server p-websvr02 10.0.0.11:80 cookie pweb2 weight 45 check
        server p-websvr03 10.0.0.115:80 cookie pweb3 weight 5 check
        server p-websvr04 10.0.0.118:80 cookie pweb4 weight 5 check

backend http_www2
        balance roundrobin
        stick-table type ip size 1m expire 6h peers prodHAproxypeers
        stick on src
        ###This site uses host headers so this type of check is required###
        option httpchk HEAD /default.htm HTTP/1.1\r\nHost:\ www2.domainname.com
        server p-websvr01 10.0.0.10:80 cookie pweb1 weight 45 check
        server p-websvr02 10.0.0.11:80 cookie pweb2 weight 45 check
        server p-websvr03 10.0.0.113:80 cookie pweb3 weight 5 check
        server p-websvr04 10.0.0.116:80 cookie pweb4 weight 5 check

backend http_www3
        balance roundrobin
        cookie SRV_ID prefix
        stick-table type ip size 1m expire 6h peers prodHAproxypeers
        stick on src
        ###This site does not use host header - only the page name is needed###
        option httpchk HEAD /login.aspx HTTP/1.1\r\nHost:\ www3.domainname.com
        server p-websvr01 10.0.0.10:80 cookie pweb1 weight 45 check
        server p-websvr02 10.0.0.11:80 cookie pweb2 weight 45 check
        server p-websvr03 10.0.0.113:80 cookie pweb3 weight 5 check
        server p-websvr04 10.0.0.116:80 cookie pweb4 weight 5 check

backend http_www4
        balance roundrobin
        cookie SRV_ID prefix
        stick-table type ip size 1m expire 6h peers prodHAproxypeers
        stick on src
        ###This site uses host header so this type of check is required###
        option httpchk HEAD /default.aspx HTTP/1.1\r\nHost:\ www4.domainname.com
        server p-websvr01 10.0.0.10:80 cookie pweb1 weight 45 check
        server p-websvr02 10.0.0.11:80 cookie pweb2 weight 45 check
        server p-websvr03 10.0.0.113:80 cookie pweb3 weight 5 check
        server p-websvr04 10.0.0.116:80 cookie pweb4 weight 5 check

backend http_www5
        balance roundrobin
        cookie SRV_ID prefix
        stick-table type ip size 1m expire 6h peers prodHAproxypeers
        stick on src
        ###This site uses host header so this type of check is required###
        option httpchk HEAD /default.aspx HTTP/1.1\r\nHost:\ www5.domainname.com
        server p-websvr01 10.0.0.10:80 cookie pweb1 weight 45 check
        server p-websvr02 10.0.0.11:80 cookie pweb2 weight 45 check
        server p-websvr03 10.0.0.115:80 cookie pweb3 weight 5 check
        server p-websvr04 10.0.0.117:80 cookie pweb4 weight 5 check

Answer 1

my side also encounter this problem 503 NOSRV

since configure on haproxy :

    acl is_www1 hdr(host) -i www1.domainname.com

but when you access :

www1.domainname.com:888

it not match

www1.domainname.com

fix by :

you configure your haproxy domain part with port :

    acl is_www1 hdr(host) -i www1.domainname.com:888

Answer 2

It seems that the https_in_ssl frontend handles the request, but since none of the backends (http_www1, http_www2 and http_www5) are selected you get a 503 error (=no backend selected, in this case). This is because the host name isn't www1.domainname.com, www2.domainname.com or *.domainname.com. You can try and capture the Host HTTP header as explained here and see what the actual value is, and why this fails.

Not sure why you chose to use ssl_fc_sni instead of hdr(host), this seems to be a bad idea and also might be the root cause of your problem.