CODEWITHSUNDEEP
amazon-web-servicesaws-application-load-balancer
johnnyb
johnnyb

Reputation: 712

Why is the ALB Failing Health Checks on a Healthy Target?

I spent quite a lot of time trying to debug this, so I thought I'd post it in case anyone else had the same issue. I was trying to debug an ALB health check issue with Fargate. I could manually connect and see that everything was coming through. I could even connect to the Fargate instance and see that the health checks were coming through and being responded to appropriately. But the ALB kept reporting the health checks as failing.

In this specific case, I am using Tomcat as the server and Fargate as the destination, and the specific error message is "request timed out", but I think other setups (and even other error messages) conform to this case.

Upvotes: 2

Views: 6757

Answers (1)

johnnyb
johnnyb

Reputation: 712

It turned out that the only problem was, on my service, I needed to GREATLY increase HealthCheckGracePeriodSeconds. This is the amount of time that the Load Balancer will wait before it starts counting health checks against you.

It turns out that there is quite a bit of latency between what the Load Balancer is doing and what it is reporting to you. By the time I was getting the "request timed out" error, the load balancer had already decided that my machine was failing health checks, but hadn't removed it from the pool yet. So, for me, it looked like it was running correctly, and the Load Balancer was still sending health checks even though it had made the decision to remove the machine from the pool. It's just the latency between the decisions being made on the Load Balancer, when it implemented those decisions, as well as when it reports them, caused quite a bit of confusion on my end.

So, if you are having problems with targets being added to your load balancer (in my case, it was a Tomcat server), especially on Fargate, be sure to check HealthCheckGracePeriodSeconds to be sure that you are giving it enough time to start all the way up. You can set it to a ridiculously high value to make sure (I think it can go up to 67 years).

Upvotes: 4

Related Questions