CODEWITHSUNDEEP
bashkubernetes
devth
devth

Reputation: 2750

Ensure Kubernetes Deployment has completed and all pods are updated and available

The status of a deployment indicates that you can look at a deployments observedGeneration vs generation and when observedGeneration >= generation then the deployment succeeded. That's fine, but I'm interested in knowing when the new container is actually running in all of my pods, so that if I hit a service I know for sure I'm hitting a server that represents the latest deployed container.

Another tip from a K8S Slack member:

kubectl get deployments | grep <deployment-name> | sed 's/ /,/g' | cut -d ' ' -f 4

I deployed a bad image, resulting in ErrImagePull, yet the deployment still reported the correct number of 8 up-date-date replicas (available replicas was 7).

Upvotes: 17

Views: 18702

Answers (2)

Zstack
Zstack

Reputation: 4733

Just use a rollout status:

kubectl rollout status deployment/<deployment-name>

This will run in foreground, it waits and displays the status, and exits when rollout is complete on success or failure. If you're writing a shell script, then check the return code right after the command, something like this.

kubectl rollout status deployment/<deployment-name>
if [[ "$?" -ne 0 ]] then
    echo "deployment failed!"
    exit 1
fi

To even further automate your script:

deployment_name=$(kubectl get deployment -n <your namespace> | awk '!/NAME/{print $1}')  
kubectl rollout status deployment/"${deployment_name}" -n <your namespace>
if [[ "$?" -ne 0 ]] then
    echo "deployment failed!"
    #exit 1
else
    echo "deployment succeeded"
fi

If you're running in default namespace then you could leave out the -n <your namespace>. The command awk '!/NAME/{print $1}') extracts the first field (deployment name), while ignoring the first row which is the header(NAME READY UP-TO-DATE AVAILABLE AGE). If you have more than one deployment files then you could also add more regex or pattern to awk: e.g. awk '!/NAME/<pattern to parse>/{print $1}')

Upvotes: 5

Timo Reimann
Timo Reimann

Reputation: 9807

Update #2: Kubernetes 1.5 will ship with a much better version of kubectl rollout status and improve even further in 1.6, possibly replacing my custom solution/script laid out below.

Update #1: I have turned my answer into a script hosted on Github which has received a small number of improving PRs by now.

Original answer:

First of all, I believe the kubectl command you got is not correct: It replaces all white spaces by commas but then tries to get the 4th field after separating by white spaces.

In order to validate that a deployment (or upgrade thereof) made it to all pods, I think you should check whether the number of available replicas matches the number of desired replicas. That is, whether the AVAILABLE and DESIRED columns in the kubectl output are equal. While you could get the number of available replicas (the 5th column) through

kubectl get deployment nginx | tail -n +2 | awk '{print $5}'

and the number of desired replicas (2nd column) through

kubectl get deployment nginx | tail -n +2 | awk '{print $2}'

a cleaner way is to use kubectl's jsonpath output, especially if you want to take the generation requirement that the official documentation mentions into account as well.

Here's a quick bash script I wrote that expects to be given the deployment name on the command line, waits for the observed generation to become the specified one, and then waits for the available replicas to reach the number of the specified ones:

#!/bin/bash
set -o errexit
set -o pipefail
set -o nounset

deployment=

get_generation() {
  get_deployment_jsonpath '{.metadata.generation}'
}

get_observed_generation() {
  get_deployment_jsonpath '{.status.observedGeneration}'
}

get_replicas() {
  get_deployment_jsonpath '{.spec.replicas}'
}

get_available_replicas() {
  get_deployment_jsonpath '{.status.availableReplicas}'
}

get_deployment_jsonpath() {
  local readonly _jsonpath="$1"

  kubectl get deployment "${deployment}" -o "jsonpath=${_jsonpath}"
}

if [[ $# != 1 ]]; then
  echo "usage: $(basename $0) <deployment>" >&2
  exit 1
fi

readonly deployment="$1"

readonly generation=$(get_generation)
echo "waiting for specified generation ${generation} to be observed"
while [[ $(get_observed_generation) -lt ${generation} ]]; do
  sleep .5
done
echo "specified generation observed."

readonly replicas="$(get_replicas)"
echo "specified replicas: ${replicas}"

available=-1
while [[ ${available} -ne ${replicas} ]]; do
  sleep .5
  available=$(get_available_replicas)
  echo "available replicas: ${available}"
done

echo "deployment complete."

Upvotes: 40

Related Questions