CODEWITHSUNDEEP
kubernetesterraformscheduleramazon-ekskubernetes-deployment
theahura
theahura

Reputation: 433

Added new node to eks, new pods still being scheduled on old nodes

I have a terraform-managed EKS cluster. It used to have 2 nodes on it. I doubled the number of nodes (4).

I have a kubernetes_deployment resource that automatically deploys a fixed number of pods to the cluster. It was set to 20 when I had 2 nodes, and seemed evenly distributed with 10 each. I doubled that number to 40.

All of the new pods for the kubernetes deployment are being scheduled on the first 2 (original) nodes. Now the two original nodes have 20 pods each, while the 2 new nodes have 0 pods. The new nodes are up and ready to go, but I cannot get kubernetes to schedule the new pods on those new nodes.

I am unsure where to even begin searching, as I am fairly new to k8s and ops in general.

A few beginner questions that may be related:

  1. I'm reading about pod affinity, and it seems like I could tell k8s to have a pod ANTI affinity with itself within a deployment. However, I am having trouble setting up the anti-affinity rules. I see that the kubernetes_deployment resource has a scheduling argument, but I can't seem to get the syntax right.

  2. Naively it seems that the issue may be that the deployment somehow isn't aware of the new nodes. If that is the case, how could I reboot the entire deployment (without taking down the already-running pods)?

  3. Is there a cluster level scheduler that I need to set? I was under the impression that the default does round robin, which doesn't seem to be happening at the node level.

EDIT: The EKS terraform module node_groups submodule has fields for desired/min/max_capacity. To increase my worker nodes, I just increased those numbers. The change is reflected in the aws eks console.

Upvotes: 1

Views: 943

Answers (1)

Jonathan
Jonathan

Reputation: 830

Check a couple of things:

  1. Do your nodes show up correctly in the output of kubectl get nodes -o wide and do they have a state of ready?
  2. Instead of pod affinity look into pod topology spread constraints. Anti affinity will not work with multiple pods.

Upvotes: 1

Related Questions