Google cloud load balancer causing error 502 - failed_to_pick_backend

Question

I've got an error 502 when I use google cloud balancer with CDN, the thing is, I am pretty sure I must have done something wrong setting up the load balancer because when I remove the load balancer, my website runs just fine.

This is how I configure my load balancer here

• Should I use HTTP or HTTPS healthcheck, because when I set up HTTPS healthcheck, my website was up for a bit and then it down again
• I have checked this link, they seem to have the same problem but it is not working for me.
• I have followed a tutorial from openlitespeed forum to set Keep-Alive Timeout (secs) = 60s in server admin panel and configure instance to accepts long-lived connections ,still not working for me.
• I have added these 2 firewall rules following this google cloud link to allow google health check ip but still didn’t work: https://cloud.google.com/load-balancing/docs/health-checks#fw-netlb https://cloud.google.com/load-balancing/docs/https/ext-http-lb-simple#firewall

When checking load balancer log message, it shows an error saying failed_to_pick_backend . I have tried to re-configure load balancer but it didn't help.

I just started to learn Google Cloud and my knowledge is really limited, it would be greatly appreciated if someone could show me step by step how to solve this issue. Thank you!

---

Answer 1

Those experienced with Kubernetes from other platforms may be confused as to why their Ingresses are calling their backends "UNHEALTHY". Health checks are not the same thing as Readiness Probes and Liveness Probes.

Health checks are an independent utility used by GCP's Load Balancers and perform the exact same function, but are defined elsewhere. Failures here will lead to 502 errors. https://console.cloud.google.com/compute/healthChecks

Answer 2

Posting an answer - based on OP's finding to improve user experience.

Solution to the error 502 - failed_to_pick_backend was changing Load Balancer from HTTP to TCP protocol and at the same type changing health check from HTTP to TCP also.

After that LB passes through all incoming connections as it should and the error dissapeared.

Here's some more info about various types of health checks and how to chose correct one.

Answer 3

The error message that you're facing it's "failed_to_pick_backend". This error message means that HTTP responses code are generated when a GFE was not able to establish a connection to a backend instance or was not able to identify a viable backend instance to connect to

I noticed in the image that your health-check failed causing the aforementioned error messages, this Health Check failing behavior could be due to:

• Web server software not running on backend instance

• Web server software misconfigured on backend instance

• Server resources exhausted and not accepting connections:

    - CPU usage too high to respond
    - Memory usage too high, process killed or can't malloc()
    - Maximum amount of workers spawned and all are busy (think mpm_prefork in Apache)
    - Maximum established TCP connections
  

Check if the running services were responding with a 200 (OK) to the Health Check probes and Verify your Backend Service timeout. The Backend Service timeout works together with the configured Health Check values to define the amount of time an instance has to respond before being considered unhealthy.

Additionally, You can see this troubleshooting guide to face some error messages (Including this).