Azure Application Gateway: Cannot connect to backend server in

Question

Due to a recent layoff I got bombarded to azure admin out of the blue. I am pretty new to this and haven't yet got the chance to follow an admin course.

Facing the following issue: We host a couple of websites on an Azure Windows Server VM running IIS. These are accessible through an application gateway with a public IP. I was asked to add two new listeners for a new part of the website. I created the appropriate targeting in the backend pool, created http and https settings and added the listeners and Rules. However, when browsing to the site, it throws a 502 error and when i check the backend health, it gives below error.

Cannot connect to backend server. Check whether any NSG/UDR/Firewall is blocking access to server. Check if application is running on correct port.

I opened up the appropriate inbound ports on the NSG of the AZ Web interface on the VM and also on the local firewall of the server hosting IIS. AFAIK there are no additional NSG rules on the application gateway.

What am i missing here? :s

Answer 1

In my case the backend pool was a RHEL 8 which had the firewall blocking all incoming connections by default. I used sudo firewall-cmd --add-port=8080/tcp --permanent in the machine without changing anything in the application gateway configs

Answer 2

In my case, I had to change the Backend Pool target from FDQN to IP to get a more specific error message, which was: "Backend certificate is invalid. Current date is not within the "Valid from" and "Valid to" date range on the certificate."

And that was because of a missing SSL Certificate binding in the server: 0.0.0.0:443, in my case.

List them with: netsh http show sslcert

Answer 3

If you've checked your firewall issues and it's not solving the problem it could be user permissions on your VM.

I then ran the following command in ps and it sorted it for me.

** netsh http add urlacl url="http://*:{port}/" user="Everyone" **

A good test to see if this will work is if you can access your app using invoke-webrequest and using a localhost URI, but not using the server's NIC Private IP.

You'll also need to set your host address to use the wildcard in your config file.

Answer 4

I have extensive experience working with Application Gateways and I can tell you that a 502 Bad Gateway means something is definitely wrong at the backend or misconfigured AGW settings - that's what the error says, so nothing surprising. From my experience here are different scenarios I've faced for this error:

1. Backend server can't be reached due to an NSG Rule controlling access from the AGW subnet to the backend subnet.
2. Backend server can be reached but the port is not opened at the server's firewall.
3. Backend server can be reached, port is opened but application is not listening on those ports or application is not even running.
4. AGW listeners were misconfigured.

Here's what you can try:

First validate whether the Application and VM are fine by trying to access the application from another VM in the same subnet.

Next, try to get a VM in a different subnet and try to access the application, to mimic the AGW trying to connect to the backend. This will help you validate whether your NSGs are properly configured.

Lastly, revisit all the AGW settings and look for any misconfiguration in the listeners or other settings. (Added this based on your comments).

Taking this approach to troubleshooting will quickly help you identify which layer is causing the issue. Also, it would be a good practice to start documenting all AGW errors you get along your journey and also the remediation steps etc. This will help you tremendously in the future - this is not the last time you'll face issues with your AGW!