CODEWITHSUNDEEP
kubernetesamazon-ekscni
dippynark
dippynark

Reputation: 3003

How can I ensure I annotate each Kubernetes node before Pods are scheduled to it on EKS?

I am trying to use CNI Custom Networking on EKS to make sure that Pod IPs are allocated from alternative subsets (to prevent IP starvation in the subnets my cluster nodes are running in). To do this I need to create some ENIConfigs and annotate each node.

How can I ensure that each node is annotated before any Pods are scheduled to it to ensure no Pod IPs are allocated from the subnets my nodes are running in?

EDIT: The only solution I can think of so far is:

However, if the above is the only workaround that is a lot of effort for a managed service

Upvotes: 2

Views: 903

Answers (1)

richardw
richardw

Reputation: 768

How about:

  • Add a ENIConfigComplete: false taint to all nodes by default
  • Deploy DaemonSet that tolerates ENIConfigComplete: false
  • DaemonSet creates a pod on each new node which
  • creates some ENIConfigs on the node (bash script??)
  • annotates each node with ENIConfigComplete: true
  • DaemonSet no longer tolerates the node, so
  • Pod is removed from the node.

The DaemonSet would ensure that every new node was properly set up.

Salesforce talk about this technique for provisioning the disks on their new nodes:

It would avoid having a long running controller process.

Upvotes: 1

Related Questions