CODEWITHSUNDEEP
kubernetes
sobi3ch
sobi3ch

Reputation: 2823

How check all my deployments in Kubernetes are finished

I have several deployments that consist on my application. I would like to perform an custom action on the end of successful deployment of my app, this equal all deployments went well. How can I determine all my kubernetes deployments finished successfully?

Upvotes: 0

Views: 1065

Answers (4)

CoatedMoose
CoatedMoose

Reputation: 4218

The cluster I work with runs several applications, and each can deploy independently, so checking for any unavailable/in progress deployments doesn't work.

Our approach assumes some things about the release process:

  • We avoid race conditions
    • We run our application releases with a CI/CD tool. The tooling reserves the environment for the application (multiple applications can deploy at the same time, but a single application won't have mulitple simultaneous deployments).
    • Each application "owns" their own resources (releases of different applications will not make changes to the same resource).
  • The K8S pods have readiness probes/health checks configured that only mark deployments "good" when they are running correctly.
  • The spec.progressDeadlineSeconds is set appropriately for each Deployment in the release

Taking an example of an application release with a web service worker, and a background worker, our deployment script might look like this:

#/bin/bash

# Cause deployment script to exit immediately if there is any failure, presumably in the `wait` commands
set -e

# This could be any action that generates kubernetes deployment(s)
kubectl apply -k "$DEPLOYMENT_DIR"

# Monitor the status of the web service worker
kubectl rollout status deployment.apps/app1-service &
_service=$!
# Monitor the status of the background worker deployment
kubectl rollout status deployment.apps/app1-background &
_background=$!

# Wait for all deployments to complete (or any of them to fail)
wait $_service
wait $_background

echo "Deployment complete"

The rollout status checks and corresponding wait actions should match the Deployments/DaemonSets/etc to be monitored as part of a release.

Upvotes: 0

Yonah Dissen
Yonah Dissen

Reputation: 1305

If all the deployments are in the same namespace you can run something like this:

 #!/bin/bash
    

check_unavailable() {
  deployments=$(kubectl get deploy -n "${NAMESPACE}" -o=jsonpath="{range .items[*]}{.metadata.name}{'\t'}{.status.unavailableReplicas}{'\n'}{end}")
  while read -r line; do lines+=("$line"); done <<<"$deployments"
  for line in "${lines[@]}"
  do
    read -ra arr <<< "$line"
    if [ "${arr[1]}" ]; then
      echo "${arr[0]} deployment has ${arr[1]} unavailable Replicas "

    fi
  done
}

check_unavailable

This should also work with deployments that have multi-container pods.

Upvotes: 0

Crou
Crou

Reputation: 11388

It's not exactly clear what do You mean.

As pointed by @Marc ABOUCHACRA, you can use watch. This can be done in more k8s way by using -w flag like so kubectl get deployments -w.

But this only provides the information about the state Desired/Current/Up-to-date/Available.

$ kubectl get deployments
NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
nginx-1   1         1         1            1           27h

You might Define Readiness probe.

Sometimes, applications are temporarily unable to serve traffic. For example, an application might need to load large data or configuration files during startup, or depend on external services after startup. In such cases, you don’t want to kill the application, but you don’t want to send it requests either. Kubernetes provides readiness probes to detect and mitigate these situations. A pod with containers reporting that they are not ready does not receive traffic through Kubernetes Services.

Note: Readiness probes runs on the container during its whole lifecycle.

readinessProbe:
  exec:
    command:
    - cat
    - /tmp/healthy
  initialDelaySeconds: 5
  periodSeconds: 5

Or just use Deployment status,

A Deployment enters various states during its lifecycle. It can be progressing while rolling out a new ReplicaSet, it can be complete, or it can fail to progress.

$ kubectl rollout status deployment nginx-1
deployment "nginx-1" successfully rolled out

Upvotes: 1

Marc ABOUCHACRA
Marc ABOUCHACRA

Reputation: 3463

Maybe with a basic watch command on all deployments ? 

watch kubectl get deployments

And check the READY column.

Or am I missing the point here ?

Upvotes: 1

Related Questions