How can I troubleshoot an AWS Application Load Balancer giving 504, while the EC2 instance behind it gives 200?

Question

I have an EC2 instance with a few applications successfully deployed onto it, listening for connections on ports 3000/3001/3002. I can correctly load a web page from it by connecting to its public DNS or public IP on the given port. I.e. curl http://<ec2-ip-address>:3000 works. So I know that the apps are running, and I know that the port bindings/firewall rules/EC2 security groups are all set up correctly to receive connections from the outside world.

I also have an Application Load Balancer, which is supposed to route traffic to the 3 apps depending on the host name, but it always gives me "504 Gateway Time-out". I've checked all the settings but I can't see what's wrong and I'm not really sure how to troubleshoot it from here.

• The ALB has a single HTTPS/443 listener, with a cert that's valid for mydomain.com, app1.mydomain.com, app2.mydomain.com, app2.mydomain.com.
• The listener has 3 rules, plus the default rule:
  1. Host == app1.mydomain.com => app1-target-group
  2. Host == app2.mydomain.com => app2-target-group
  3. Host == app3.mydomain.com => app3-target-group
  4. Default action (last resort) => default-target-group
• Each target group contains only the single EC2 instance, over HTTP, with the following ports:
  1. app1-target-group: 3000
  2. app2-target-group: 3001
  3. app3-target-group: 3002
  4. default-target-group: 3000

Given that I can access the app directly, I'm sure it must be a problem with the way I've configured the ALB/listener/target groups. But the 504 doesn't give me much to go on.

I've tried to turn on access logs to an S3 bucket, but it doesn't seem to be writing anything there. There's a single object called ELBAccessLogTestFile, and no actual logs in the bucket.

EDIT: Some more information... I actually have nginx installed on the EC2 instance, which is where I was previously doing the SSL termination and hostname-to-port mapping/routing. If I change the default-target-group above to point to port 443 over HTTPS, then it works!

So for some reason, routing traffic - from the ALB to the EC2 instance over HTTPS on port 443 -> OK! - from the ALB to the EC2 instance over HTTP on port 3000 -> Broken!

But again, I can hit the instance directly on HTTP/3000 from my laptop.

Answer 1

Communication between resources in the same security group is not open by default. Security group membership alone does not provide special access. You still need to open the ports in the security group to allow other resources in the security group to access those ports. You can specify the security group ID in the rule's source field if you don't want to open it up beyond the resources in the security group.