CODEWITHSUNDEEP
amazon-web-serviceskubernetesamazon-ec2amazon-ekskubernetes-cronjob
robliv
robliv

Reputation: 1541

Prevent cluster autoscaling from killing pod

I have a AWS EKS K8s setup with autoscaling group that uses spot instances.

Once a day there is a scheduled K8s cronjob which scales down a bunch of deployments, runs a script in a pod for 15min , then scales the deployments back up.

Issue is after deployments are scaled down, the script starts running and cluster autoscaling triggers to remove some nodes around 10min later which kills the script pod and moves it to different node.

This can't happen, how do I configure the cronjob script pod so that it never gets killed by autoscaler?

Thanks!

Upvotes: 0

Views: 1574

Answers (1)

Harsh Manvar
Harsh Manvar

Reputation: 30160

This can't happen, how do I configure the cronjob script pod so that it never gets killed by autoscaler?

You can use the Node or POD affinity for scheduling the POD on specific Node so cronjob will not runt he PODs on Spot instances only you stateless workload will run on spot instance about which you don't care much.

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: test
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          affinity:
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                  - key: app
                    operator: In
                    values:
                    - web-store
                topologyKey: "kubernetes.io/hostname"
          containers:
            - name: hello
              image: bash
              command: ["echo",  "Hello world"]
          restartPolicy: OnFailure

You can use the Node selector also to schedule the POD on specific type of Node group which is not scaling up & down.

Node affinity example

affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/e2e-az-name
            operator: In
            values:
            - e2e-az1
            - e2e-az2

Using the affinity you can plan on which node to schedule your POds.

Read more at : https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/

Upvotes: 2

Related Questions