EKS Kubernetes outbound traffic

Question

I have recently setup a EKS cluster in AWS for my company's new project. Before I get into my issue, here is some info of my setup. There are two nodes (at the moment) in the nodegroup with auto-scaling. At first, I created the nodes in the private subnet as I think it is more secure. But my supervisor told me that we will need the capability to SSH to the node directly, so I recreated the nodes in the public subnet so that we can SSH to the node using public key.

We have two CMS instances sitting in AWS (for example aws.example.com) and DigitalOcean (for example do.example.com) that contains some data. When the containers in EKS cluster start, some of them will need to access the instance in AWS by using the url aws.example.com or do.example.com. If the containers in EKS failed to access the instances, the container will still run but the app in it won't. So I need to whitelist the public IP of all my EKS nodes on the two CMS instances in order for the app work. And it works.

I am using ingress in my EKS cluster. When I created the ingress object, the AWS created an application load balancer for me. All the inbound traffic is being handled by the ALB, it is great. But here comes the issue. When more containers are created, the auto-scaling spin up new nodes in my EKS cluster (with different public IP every time), then I will have to go to the two CMS instances to whitelist the new public IP address.

So, is there any way to configure in such a way that all the nodes to use a single fixed IP address for outbound traffic? or maybe configure them to use the ALB created by ingress for outbound traffic as well? or I need to create a server to do that? I am very lost right now.

Here is what I have tried: When the cluster is created, it seems like it created a private subnet as well even though I specify the nodes to be created in public subnet. There is a nat-gateway (ngw-xxxxxx) created for the private subnet and it comes with an Elastic IP (for example 1.2.3.4). The routetable of the public subnet is as below: 192.168.0.0/16 local 0.0.0.0/0 igw-xxxxxx

So I thought by changing igw-xxxxxx to ngw-xxxxxx, all the outbound traffic will use the ngw-xxxxxx and send the traffic to the outside world using IP address 1.2.3.4, which I just need to whitelist 1.2.3.4 on my two CMS instances. But right after I applied the changes, all containers are terminated and all things stopped working.

Answer 1

Exactly, as @Marcin mentioned in the comment above. You should move your node-group to the private subnet.

Private subnet

As the docs tell:

Use a public subnet for resources that must be connected to the internet, and a private subnet for resources that won't be connected to the internet

The idea of private subnet is to forbid access to resources inside directly from the internet.

You can read really good part of AWS documentation here: https://docs.aws.amazon.com/vpc/latest/userguide/how-it-works.html

For private subnet you need to setup outgoing traffic thru your Nat gateway in the Route Table (read here: https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html).

Public subnet

If you need your cluster in the public subnet for some reasons, but it is BAD parctice, you can do the trick:

You can route traffic via Nat Gateways from public subnet only to a specific server(your CMS).

Your public subnet route table may look like:

Destination	Target
10.0.0.0/16	local
W.X.Y.Z/32	nat-gateway-id
0.0.0.0/0	igw-id

Where W.X.Y.Z/32 is your CMS IP address.

---

Some hint

Moreover good practice is to allocate a pool of EIP and attach them to NAT Gateway to be sure it is not changed in the future.

When you want to modify infrastructure and create more complicated Nat (e.g. you want to filtering traffic on layer 7), you can createHigh Availability Nat instances, and attach the IP to NAT instances instead of NAT Gateways.

In that situation you will avoid mailing some 3-rd party APIs to whitelist your new IPs.