CODEWITHSUNDEEP
kubernetes
anon
anon

Reputation:

How to wait for a rollout to complete in Kubernetes

As part of our CI pipeline, we have a deployment script for a number of web services that looks something like this:

kubectl apply -f deployment1.yml
kubectl apply -f deployment2.yml

The problem is that the next stage of the pipeline sometimes fails because the services are not ready by the time it starts.

I would like to add a line in the script that says something like:

Wait until all deployments are in the Ready state, or fail if more than 30 seconds has elapsed.

I thought that the following would work but unfortunately it seems that the timeout flag is not available:

kubectl rollout status deployment deployment1 --timeout=30s
kubectl rollout status deployment deployment2 --timeout=30s

I don't want to run "kubectl rollout status" without a timeout as that will cause our build to hang if there is a failure in one of the deployments.

Upvotes: 18

Views: 38371

Answers (3)

Stefan Lasiewski
Stefan Lasiewski

Reputation: 18331

Recent versions of kubectl do support a timeout option:

$ kubectl create -f ds-overlaytest.yml
daemonset.apps/overlaytest created
$ kubectl rollout status ds/overlaytest --timeout=10s
Waiting for daemon set spec update to be observed...
error: timed out waiting for the condition
$

Check out the kubectl reference on how to use this option: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#-em-status-em-

Upvotes: 15

Alex Spurling
Alex Spurling

Reputation: 55864

I found a solution that works well. Set the property .spec.progressDeadlineSeconds to a value such as 30 (default is 600 or ten minutes) and kubectl rollout status deployment will wait for this amount of time before displaying an error message and exiting with a non zero exit code:

$ kubectl rollout status deploy/nginx
Waiting for rollout to finish: 2 out of 3 new replicas have been updated...
error: deployment "nginx" exceeded its progress deadline

Documentation is here: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#failed-deployment

Upvotes: 15

Spazzy757
Spazzy757

Reputation: 929

You could possibly handle this with pure bash i.e.:

ATTEMPTS=0
ROLLOUT_STATUS_CMD="kubectl rollout status deployment/myapp -n namespace"
until $ROLLOUT_STATUS_CMD || [ $ATTEMPTS -eq 60 ]; do
  $ROLLOUT_STATUS_CMD
  ATTEMPTS=$((attempts + 1))
  sleep 10
done

This is specified in this blog

However, I do not believe there is a Kubernetes Native way to wait for a deployments rollout, you could possibly accheive this with hooks in Helm or Webhooks if you want to get really fancy

Upvotes: 1

Related Questions