CODEWITHSUNDEEP
bashkubernetes
Abdelwahhab
Abdelwahhab

Reputation: 83

How to list kubernetes pods with age less than a specific period

I am writing a script that runs each minute and looks for newly created pods. The script executes commands within these pods.

I need to add the missing part to my current solution which looks like this:

while true; do
  pods= $(kubectl get pods --field-selector status.phase=='Running' ---------- someting should be here ----------)
    for pod in ${pods[@]} ; do
    actions
    done
  sleep 60;
done

Upvotes: 2

Views: 3363

Answers (2)

Abdelwahhab
Abdelwahhab

Reputation: 83

My current solution looks like this. Please feel free to review and enhance.

current_pods=$(kubectl get pods | grep "Running" | awk '{print $1}')
# attach existing pods    
for pod in $current_pods; do
  do-something
done
# attach newly created pods each 60s
while true; do
  new_pods=$(kubectl get pods | grep "Running" | awk '{print $1}')
    for pod in $new_pods; do
      if [[ ! " ${current_pods[@]} " =~ $pod ]]; then
        do-something
      fi
    done
  current_pods=$new_pods
  sleep 60;
done

Upvotes: 1

Olesya Bolobova
Olesya Bolobova

Reputation: 1663

I'd suggest to make use of Kubernetes watch and/or events APIs.
Concrete implementation depends on what events you are actually interested in.
What do you mean by "newly created pods"?

This command will stream you the names of new pods entering the Running phase.

# --watch-only can be replaced by --watch
# --watch will show all current pods in Running state + stream changes
# --watch-only will just stream changes
kubectl get pods --field-selector status.phase=='Running' \
  --watch-only -ojsonpath='{.involvedObject.name}{"\n"}'

But this might be not what you want.
"Running" phase refers to a pod state, not to a state of related containers. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase
As a result you can get the following output.

# from API server point of view all these pods are in Running phase,
# regardless of some containers failing
kubectl get pods --field-selector status.phase=Running --watch-only

NAME         READY   STATUS             RESTARTS   AGE
pod1         2/2     Running            30         31h
pod2         1/2     Running            30         31h
pod2         1/2     Error              30         31h
pod2         1/2     CrashLoopBackOff   30         30h

This command will stream you the names of newly scheduled pods (i. e. scheduling decision has been dispatched to a kubelet, but pod's containers may not be running yet).
This can be viewed as a moment of actual pod object creation, but doesn't say anything about whether the pod conainers have succesfully started.

kubectl get events --field-selector involvedObject.kind=Pod,reason=Scheduled \
  --watch-only -ojsonpath='{.involvedObject.name}{"\n"}'

This command will stream you the names of new pods which have started successfully.
Technically it means that certain conditions are met, like "all containers in the Pod are ready" and "the Pod is able to serve requests and should be added to the load balancing pools of all matching Services".
https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions

# as above: you can replace --watch-only by --watch
# --watch will show the pods that are already running successfully + stream newcomers
# --watch-only will just stream newcomers
kubectl get pods --field-selector status.phase=Running --watch-only -o json | \
  jq '. | select (.status.conditions[] | select (.type=="Ready" and .status=="True")) | .metadata.name'

Using these examples you could construct necessary conditions according to your needs and run a simple command like this.

# kubectl watch will timeout in a period of time
# timeout value is configured by API server param
# --min-request-timeout [default=1800s]
# also if API server restarts, the connection will be dropped
# so we still need 'while true' here
while true; do
  kubectl get events --field-selector involvedObject.kind=Pod,reason=Scheduled \
    --watch-only -ojsonpath='{.involvedObject.name}{"\n"}' | \
  while read pod; do
    echo "$pod"
  done
done

Frankly this approach doesn't guarantee you 100% consistency. You may miss some events with --watch-only (say, when the client restarts), as well as you may have duplicates with --watch.
For simple usecases keeping a short-lived vocabulary of proccessed pods is probably fine.

For more sophisticated tasks you could turn to advanced usage of watch APIs.
There is a good article to start with.
https://learnk8s.io/real-time-dashboard

Upvotes: 0

Related Questions