Haproxy & Tomcat results in random 502 Bad Gateway

Question

We are using Tomcat 8 with HAProxy 1.5.4. We are getting random 502 Bad Gateway errors from Haproxy. When we access Tomcat directly the error does not manifest, so it must be something with how Haproxy is configured.

Any pointers would be really appreciated! Please find below the error and the config files. Thank you!

The error log:

Mar 1 11:41:37 www1 haproxy[15362]: xx.xx.xx.xx:56387 [01/Mar/2016:11:41:35.480] https-in~ servers/www1a 1987/0/0/-1/2029 502 8878 - - PH-- 1764/1758/46/26/0 0/0 "POST /abc/test/b25766378a05446496645649e2ddaf7a/poll HTTP/1.1"

Tomcat HTTP connector configuration

<Connector 
    URIEncoding             = "UTF-8"
    port                    = "8080" 
    protocol                = "HTTP/1.1" 
    maxThreads              = "1850"
    connectionTimeout           = "900000"
    keepAliveTimeout            = "900000"
    maxKeepAliveRequests            = "-1"/>

Haproxy configuration

global
   log /dev/log local0
   log /dev/log local1 notice
   chroot /var/lib/haproxy
   stats socket /run/haproxy/admin.sock mode 777 level admin
   stats timeout 30s
   user haproxy
   group haproxy
   daemon

   # Per process limit: The default is 2000, too small for us
   maxconn 18000

   # Increase the cache from 20000 (default), higher values reduce CPU usage
   tune.ssl.cachesize 60000

   # Default SSL material locations
   ca-base /etc/ssl/certs
   crt-base /etc/ssl/private

   # Default ciphers to use on SSL-enabled listening sockets.
   # For more information, see ciphers(1SSL).
   ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL
   ssl-default-bind-options no-sslv3 no-tls-tickets

defaults
   log  global
   mode http
   option   httplog
   option  http-server-close
   option  forwardfor
   option   dontlognull

   # Set the listen limit: The default is 2000, too small for us
   maxconn 9000

   # we should fix this
   option accept-invalid-http-response
   option accept-invalid-http-request
   no option checkcache

   timeout connect 80000
   timeout client  900000
   timeout server  500000

 frontend http-in
   bind *:80


    # Redirect all subdomains to www.
    redirect prefix https://www.example.com code 301 if !{ hdr_beg(host) -i     www. }

    # Redirect all trafic to https 
    redirect scheme https if !skip_pages !{ ssl_fc }

    default_backend servers

  frontend https-in

    # add no-tlsv10 for disabling tls 1.0
    bind *:443 ssl  crt /etc/ssl/private/www_example_com.pem


    default_backend servers

    # Redirect all subdomains to www.
    redirect prefix https://www.example.com code 301 if !{ hdr_beg(host) -i www. }

    backend servers

    # Every connection is closed and opened to the server
    option http-server-close

    # Recommended to enable
    option http-pretend-keepalive

    # The url to check the backend servers health
    option httpchk GET /srvstatus.htm

    # Balancing
    balance roundrobin
    appsession JSESSIONID len 52 timeout 3h request-learn prefix
    stick-table type string len 32 size 1M expire 3h

    # We have 3 backend servers, one is for backup
    server www1a 127.0.0.1:8080 check
    server www2a xx.xx.xx.xx:8080 check
    server www1b 127.0.0.1:8081 check  backup

Answer 1

I have been searching for a very similar situation where some, not all, requests from a Geoserver backend service was returned as 500 Internal server error and the Geoserver has been working flawlessly in the passed. HAProxy logged this as PH-- in its log. Geoserver had nothing in its logs.

For me it turned out that my over eager use of headers for debug purpose was the culprit and after reducing/removing the majority of them the problem stopped.

Reading @Robertos answer I believe that this could also be a solution to my problem.

I know this is not an answer to the original question but since I have googled for some time (days) without finding this answer until just now I would like to add this so others can find this answer.

HAProxy documentation only states that it returns PH-- when the backend HTTP response doesn't comply to standards which is clearly wrong. It also does this when the buffer is overrun or exhausted.

Answer 2

Unfortunately Robertos answer didn't help me, but reading my haproxy logs they very aggressively ask you to set timeouts, so in my desperation I did, like this:

defaults
    timeout connect 10s
    timeout client 30s
    timeout server 30s

Now haproxy with tomcat behind doesn't return any 502 anymore.

I do not understand why, as default values seem to be 3s and my latency is <50 ms.

Answer 3

I had the same problem with tomcat and haproxy. Tomcat was responding a http 200 response, but haproxy was proxying it and responding with 502. The cause was in that haproxy limits:

   tune.bufsize 1638400
   tune.http.maxhdr 10000

I changed this two properties to a high value. The default value is restrictive and transforms http 200 response from backend into 502 response. BE CAREFUL with the high values of this two properties because you can easily have a memory problem. Once I changed this properties, the problem with 502 was over.