AWS Network Load Balancer without Target in every AZ

Question

In the AWS Network Load Balancer documentation it says that when specifying instances for a Target Group that it must include an instance in every AZ that the Load Balancer is registered in. This is not enforced.

What happens to traffic if you have an NLB registered in 3 AZs, but only a single target EC2 instance in AZ1? What if you enable cross AZ load balancing, does that make any difference?

Answer 1

Your scenario wouldn't require cross-zone load balancing be enabled. As Marcin pointed it, it does nothing for you. In fact, resolve your NLB's DNS and you'll see that it only returns an A record for each AZ that has a healthy instance aggregated in all of the NLB's target gouups. Marcin's response is fantastic for the deep dive.

Some folks are here saying "yes, but we're still getting timeouts.". This is because your scenario is more complex that OP's. In short, you likely have an NLB with more than one target group in which distinct targets exist in multiple AZs. Turning on cross-zone load balancing will solve your suboptimal configuration at the cost of additional data transfer charges on your bill. Duck tape and gum wrappers work in AWS VPC.

More info:

NLBs are "smart" with DNS in that their VIP will only resolve to A records that have healthy targets (within a timely reason). If an NLB has several target groups with distinct instances across three AZs, you'll be getting three A records returned (one for each AZ that has a healthy target). This is how NLB + RR DNS works.

However, if your target groups contains EC2 instance(s) in a single AZ then it's a 33% (given three AZs) chance that the DNS round robin will resolve the proper AZ.

The best solution is to turn on cross-zone load balancing. This increases data transfer cost, but it is less complex to the alternative of breaking out the NLBs. Please note that enabling cross-zone load balancing will take a few minutes to kick in. Don't enable it, kick off a telnet immediately and be sad when it doesn't work. Wait 5 - 10 minutes and then kick off your telnet.

Source: Anecdotal and practical experience working with AWS and janky EC2 solutions.

Answer 2

What happens to traffic if you have an NLB registered in 3 AZs, but only a single target EC2 instance in AZ1? What if you enable cross AZ load balancing, does that make any difference?

In this particular scenario (NLB in 3 AZs, and single instance in 1 AZ), nothing really happens. There is no apparent difference with, or without, cross-zone load balancing from the perspective of the end-user. The instance will be accessible in either case.

To verity that, I developed a simple CloudFormation template the creates NLB, with, or without cross-zone load balancing, and 1 instance. The template allows for easy experimentation with different setups of NLB, cross-zone and instance location. I used the template in us-east-1 region and default VPC.

For the template you specify several parameters, including:

• NLBSubnetsIds - subnets where to enable NLB. You have to check first in console, which subnets are in which AZs.

• InstanceSubnetId - subnet for the instance. Again you can check which subnet is in which AZ if you want play around with instance location. You must ensure that instance is created in one of AZs set for your NLB.

• CrossZoneEnabled - enable or disable cross-zone balancing for the NLB.

Once you create the stack from the template and instance health check pass (can take 1 or 2 minutes), you can access NLB DNS in your browser to view a sample webpage hosted on the instance.

---

Parameters:

  VpcId:
    Type: AWS::EC2::VPC::Id
    
  NLBSubnetsIds:
    Type: List<AWS::EC2::Subnet::Id>
    
  InstanceSubnetId:
    Type: AWS::EC2::Subnet::Id 
    
  AmazonLinux2AMIId:
    Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
    Default: /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2
    
  CrossZoneEnabled:
    Type: String  
    Default: false
    AllowedValues: [true, false]

Resources:


  BasicSecurityGroup:                                                        
      Type: AWS::EC2::SecurityGroup                                          
      Properties: 
        GroupDescription: Enable www port
        SecurityGroupIngress:
          - IpProtocol: tcp
            FromPort: 80
            ToPort: 80
            CidrIp: 0.0.0.0/0            
        VpcId:  !Ref VpcId

  MyInstance1:
    Type: AWS::EC2::Instance

    CreationPolicy:
        ResourceSignal:
          Timeout: PT5M
                
    Properties:                
      ImageId: !Ref AmazonLinux2AMIId  
      InstanceType: t2.micro        
      Monitoring: false
      SecurityGroupIds: [!Ref BasicSecurityGroup]
      SubnetId: !Ref InstanceSubnetId
      UserData: 
        Fn::Base64: !Sub |
            #!/bin/bash -xe

            yum install -y httpd aws-cfn-bootstrap

            echo "<h2>Hello world from $(hostname -f)</h2>" \
              > /var/www/html/index.html

            systemctl start httpd

            # check if website is working
            curl -s localhost | grep "Hello"

            # Signal the status from cfn-init
            /opt/aws/bin/cfn-signal -e $? \
                --stack ${AWS::StackName} \
                --resource MyInstance1 \
                --region ${AWS::Region}
                
                
  MyNLB:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties: 
      IpAddressType: ipv4
      LoadBalancerAttributes:  
        - Key: load_balancing.cross_zone.enabled
          Value: !Ref CrossZoneEnabled
      Scheme: internet-facing 
      Subnets: !Ref NLBSubnetsIds
      Type: network
      
  MyListner1:      
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties: 
      DefaultActions: 
        - TargetGroupArn: !Ref MyTargetGroup
          Type: forward 
      LoadBalancerArn: !Ref MyNLB
      Port: 80 
      Protocol: TCP 

  MyTargetGroup: 
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties: 
      HealthCheckEnabled: true
      HealthCheckIntervalSeconds: 10
      HealthCheckPath: /
      HealthCheckProtocol: HTTP 
      HealthyThresholdCount: 2
      UnhealthyThresholdCount: 2
      Port: 80
      Protocol: TCP 
      TargetGroupAttributes: 
        - Key: deregistration_delay.timeout_seconds
          Value: 30
      Targets:
        - Id: !Ref MyInstance1
          Port: 80
      TargetType: instance 
      VpcId: !Ref VpcId
      
      
Outputs:
    
  DNSName:
    Value: !GetAtt MyNLB.DNSName

From the end-user perspective, in your scenario there is no clear difference between enabling or disabling cross-zone in NLB. However, the long-term difference could be in high availability. Namely, if you have cross-zone disabled and if something happens with a NLB node in the AZ where the instance is located, NLB won't be able to route traffic to your instance from other AZ. This is my speculation, as this is not something which you can check manually. The reason is that once you associate an AZ/subnet with your NLB, you can't disassociate it, to check what happens in such scenario.

In contrast, if cross-zone is enabled, in the above scenario, NLB node from other zone could probably route traffic to the instance across zones.

The major benefit of having cross-zone traffic enabled, is when you different number of instances in different AZs. In this case, cross-zone balancing enables that all instances will get roughly same amount of traffic. Without, cross-zone balancing, an isolate instance would get much more traffic then the collection of instances in the other AZ.

You can check the effects of zone-balancing using the second template. The template almost same as before, but now 1 AZ will have 3 instances, while the other one will have 1 AZ.

---

Parameters:

  VpcId:
    Type: AWS::EC2::VPC::Id
    
  NLBSubnetsIds:
    Type: List<AWS::EC2::Subnet::Id>
    
  InstanceSubnetId1:
    Type: AWS::EC2::Subnet::Id 

  InstanceSubnetId2:
    Type: AWS::EC2::Subnet::Id     
    
  AmazonLinux2AMIId:
    Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
    Default: /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2
    
  CrossZoneEnabled:
    Type: String  
    Default: false
    AllowedValues: [true, false]

Resources:


  BasicSecurityGroup:                                                        
      Type: AWS::EC2::SecurityGroup                                          
      Properties: 
        GroupDescription: Enable www port
        SecurityGroupIngress:
          - IpProtocol: tcp
            FromPort: 80
            ToPort: 80
            CidrIp: 0.0.0.0/0            
        VpcId:  !Ref VpcId

  MyInstance1:
    Type: AWS::EC2::Instance

    CreationPolicy:
        ResourceSignal:
          Timeout: PT3M
                
    Properties:                
      ImageId: !Ref AmazonLinux2AMIId  
      InstanceType: t2.micro        
      Monitoring: false
      SecurityGroupIds: [!Ref BasicSecurityGroup]
      SubnetId: !Ref InstanceSubnetId1
      UserData: 
        Fn::Base64: !Sub |
            #!/bin/bash -xe

            yum install -y httpd aws-cfn-bootstrap

            echo "<h2>Hello world from $(hostname -f)</h2>" \
              > /var/www/html/index.html

            systemctl start httpd

            # check if website is working
            curl -s localhost | grep "Hello"

            # Signal the status from cfn-init
            /opt/aws/bin/cfn-signal -e $? \
                --stack ${AWS::StackName} \
                --resource MyInstance1 \
                --region ${AWS::Region}
                

  MyInstance2:
    Type: AWS::EC2::Instance

    CreationPolicy:
        ResourceSignal:
          Timeout: PT3M
                
    Properties:                
      ImageId: !Ref AmazonLinux2AMIId  
      InstanceType: t2.micro        
      Monitoring: false
      SecurityGroupIds: [!Ref BasicSecurityGroup]
      SubnetId: !Ref InstanceSubnetId2
      UserData: 
        Fn::Base64: !Sub |
            #!/bin/bash -xe

            yum install -y httpd aws-cfn-bootstrap

            echo "<h2>Hello2 world from $(hostname -f)</h2>" \
              > /var/www/html/index.html

            systemctl start httpd

            # check if website is working
            curl -s localhost | grep "Hello"

            # Signal the status from cfn-init
            /opt/aws/bin/cfn-signal -e $? \
                --stack ${AWS::StackName} \
                --resource MyInstance2 \
                --region ${AWS::Region}


  MyInstance3:
    Type: AWS::EC2::Instance

    CreationPolicy:
        ResourceSignal:
          Timeout: PT3M
                
    Properties:                
      ImageId: !Ref AmazonLinux2AMIId  
      InstanceType: t2.micro        
      Monitoring: false
      SecurityGroupIds: [!Ref BasicSecurityGroup]
      SubnetId: !Ref InstanceSubnetId2
      UserData: 
        Fn::Base64: !Sub |
            #!/bin/bash -xe

            yum install -y httpd aws-cfn-bootstrap

            echo "<h2>Hello2 world from $(hostname -f)</h2>" \
              > /var/www/html/index.html

            systemctl start httpd

            # check if website is working
            curl -s localhost | grep "Hello"

            # Signal the status from cfn-init
            /opt/aws/bin/cfn-signal -e $? \
                --stack ${AWS::StackName} \
                --resource MyInstance3 \
                --region ${AWS::Region}


  MyInstance4:
    Type: AWS::EC2::Instance

    CreationPolicy:
        ResourceSignal:
          Timeout: PT3M
                
    Properties:                
      ImageId: !Ref AmazonLinux2AMIId  
      InstanceType: t2.micro        
      Monitoring: false
      SecurityGroupIds: [!Ref BasicSecurityGroup]
      SubnetId: !Ref InstanceSubnetId2
      UserData: 
        Fn::Base64: !Sub |
            #!/bin/bash -xe

            yum install -y httpd aws-cfn-bootstrap

            echo "<h2>Hello2 world from $(hostname -f)</h2>" \
              > /var/www/html/index.html

            systemctl start httpd

            # check if website is working
            curl -s localhost | grep "Hello"

            # Signal the status from cfn-init
            /opt/aws/bin/cfn-signal -e $? \
                --stack ${AWS::StackName} \
                --resource MyInstance4 \
                --region ${AWS::Region}                
                
  MyNLB:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties: 
      IpAddressType: ipv4
      LoadBalancerAttributes:  
        - Key: load_balancing.cross_zone.enabled
          Value: !Ref CrossZoneEnabled
      Scheme: internet-facing 
      Subnets: !Ref NLBSubnetsIds
      Type: network
      
  MyListner1:      
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties: 
      DefaultActions: 
        - TargetGroupArn: !Ref MyTargetGroup
          Type: forward 
      LoadBalancerArn: !Ref MyNLB
      Port: 80 
      Protocol: TCP 

  MyTargetGroup: 
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties: 
      HealthCheckEnabled: true
      HealthCheckIntervalSeconds: 10
      HealthCheckPath: /
      HealthCheckProtocol: HTTP 
      HealthyThresholdCount: 2
      UnhealthyThresholdCount: 2
      Port: 80
      Protocol: TCP 
      TargetGroupAttributes: 
        - Key: deregistration_delay.timeout_seconds
          Value: 30
      Targets:
        - Id: !Ref MyInstance1
          Port: 80
        - Id: !Ref MyInstance2
          Port: 80
        - Id: !Ref MyInstance3
          Port: 80
        - Id: !Ref MyInstance4
          Port: 80                              
      TargetType: instance 
      VpcId: !Ref VpcId
      
      
Outputs:
    
  DNSName:
    Value: !GetAtt MyNLB.DNSName

If you use the above template, and repeatedly request the NLB url, you will see that the isolated instance will get about 50% of the traffic without cross-zone balancing. With cross-zone balancing enabled, it will be about 20%. Below are my results based on 100 requests: