CODEWITHSUNDEEP
nginxtomcat7http-status-code-403amazon-elbintermittent
ZekiraDrake
ZekiraDrake

Reputation: 165

Intermittent 403 when accessing server via Load Balancer, can't figure out why

Okay, so here's the scenario. We send our API Requests to an NGINX Server, which then redirects it to an AWS Elastic Load Balancer, which has targets pointing to our backend servers. The backend server processes the request, returns a response. Nothing out of the ordinary, right?

Well, for whatever reason, sometimes the POST requests coming from a specific API resource ends up with a 403. We see it in the Proxy Server logs (/var/log/nginx/access.log) that there's a 403 returned, and then Load Balancer logs (access logs, writes to S3) also say 403. However, no logs at all in the backend servers (catalina.out) saying that the request even arrived. This makes me believe that the Load Balancer is somehow discarding some of the requests and never makes it to the backend. Of course, this is just a surface level assumption. I'm really not sure where the request is getting stuck/discarded.

Something to note is that during the 403 scenario, it takes only like <60ms for our request to be returned a 403. If it returns a 200, it usually takes around ~250ms. So it seems like the Load Balancer doesn't even try to bring it to the backend server at all and just assumes a 403 somewhere.

It being intermittent just makes the problem even worse, as pinpointing the problem is even harder.

We've actually tried migrating to a modern Application Load Balancer, and for a while the problem kind of simmered down. But now we're getting more intermittent 403s again even with the updated Load Balancer.

The problem's almost a year old now, and still haven't found a solution that would put the 403 Forbidden chance to near 0%.

Completely at a loss here. Any idea would be appreciated.

Upvotes: 2

Views: 5802

Answers (1)

ZekiraDrake
ZekiraDrake

Reputation: 165

So it turns out that it was the fault of mod_security all this time. I don't know how I they missed telling me that crucial detail where mod_security was actually installed in the backend servers, and that's where the requests were getting intercepted.

We ended up whitelisting some rules on mod_security so that it doesn't aggressively disrupt some of the API calls being made from external sources.

Upvotes: 0

Related Questions