CODEWITHSUNDEEP
amazon-web-serviceskuberneteskubernetes-podamazon-eks
samxiao
samxiao

Reputation: 2667

Migrate Pods from different EC2 hosts

I am new to AWS, Kubernetes, EKS, and AppMesh, but had done DevOps in previous roles.

I am taking over some K8 cluster which used EKS and found that we set up NAT gateway which helps redirect egress traffic outbound as a single IP (we need that for whitelisting as 3rd-party external service require it). Pods hosted in a private subnet works fine.

But I found that Pods which hosted on public subnet just skip the NAT gateway, it uses the Public DNS (IPv4) IP address for outbound calls, which don't work for us as it does not use the single NAT gateway.

So I have a few questions:

  1. How do we migrate Pods from Public subnet Hosts to Private subnets Hosts?
  2. Should we use nodeSelector, Node affinity? Do labelings on the Nodes work?
  3. I am not sure why we have Nodes in a public subnet, but we followed this guide: https://docs.aws.amazon.com/eks/latest/userguide/create-public-private-vpc.html
  4. If we do choose to be on fully private subnets, can we make an exception for such mapping that we allow some Pods to have HTTP endpoints to be exposed for ingress traffic, while still on private subnets?
  5. What do you recommend us to handle when a Pod/Container needs to use NAT gateway for egress traffic, but then exposing HTTP endpoints for ingress traffic?

Note that currently, our EKS is by default set to all public, should we move to Public and private mode?

Thanks for all the answers in advance!.

Upvotes: 1

Views: 145

Answers (1)

Rico
Rico

Reputation: 61521

How do we migrate Pods from Public subnet Hosts to Private subnets Hosts? Should we use nodeSelector, Node affinity? Do labelings on the Nodes work?

Yes. Use Node Affinity which same as using a nodeSelector. You can do a rolling change by updating whatever resource you are using to manage your pods (i.e Deployment, Statefulset, DaemonSet, etc). If you configured it correctly the next time your pods start, they will be in the private subnet hosts.

I am not sure why we have Nodes in a public subnet, but we followed this guide: https://docs.aws.amazon.com/eks/latest/userguide/create-public-private-vpc.html

The guide says public subnet so it makes sense that there is one.

If we do choose to be on fully private subnets, can we make an exception for such mapping that we allow some Pods to have HTTP endpoints to be exposed for ingress traffic, while still on private subnets?

Yes! you can create an externally facing load balancer (ALB, NLB, or ELB). These can also be managed by Kubernetes if you use the Service type LoadBalancer. You'll need the appropriate annotations in your Service definitions to get what you want.

What do you recommend us to handle when a Pod/Container needs to use NAT gateway for egress traffic, but then exposing HTTP endpoints for ingress traffic?

Use an externally facing load balancer that forwards traffic to your private VPC with the Kubernetes Service type LoadBalancer and use AWS NAT Gateways for outgoing internet traffic.

Disclaimer: This is just a recommendation, there are other combinations and alternatives.

Upvotes: 2

Related Questions