CODEWITHSUNDEEP
dockerdocker-swarm
ext
ext

Reputation: 2903

Docker swarm get deployment status

After running docker stack deploy to deploy some services to swarm is there a way to programmatically test if all containers started correctly?

The purpose would be to verify in a staging CI/CD pipeline that the containers are actually running and didn't fail on startup. Restart is disabled via restart_policy.

I was looking at docker stack services, is the replicas column useful for this purpose?

$ docker stack services --format "{{.ID}} {{.Replicas}}" my-stack-name
lxoksqmag0qb 0/1
ovqqnya8ato4 0/1

Upvotes: 5

Views: 2938

Answers (5)

Patrick
Patrick

Reputation: 167

Since v26.0, docker stack deploy and docker stack rm support the --detach flag like docker service create and docker service update: https://docs.docker.com/reference/cli/docker/stack/deploy/

It defaults to true, but if you set it to false, it'll wait for the stack services to converge.

docker stack deploy --detach=false STACK

Upvotes: 2

Raphael PICCOLO
Raphael PICCOLO

Reputation: 2175

you can call this for every service. it returns when converged. (all ok)

docker service update STACK_SERVICENAME

Upvotes: -1

matt
matt

Reputation: 53

Another solution might be to use docker service scale - it will not return until service is converged to specified amount of replicas or will timeout.

export STACK=devstack # swarm stack name
export SERVICE_APP=yourservice # service name
export SCALE_APP=2 # desired amount of replicas

docker stack deploy $STACK --with-registry-auth
docker service scale ${STACK}_${SERVICE_APP}=${SCALE_APP}

One drawback of that method is that you need to provide service names and their replica counts (but these can be extracted from compose spec file using jq).

Also, in my use case I had to specify timeout by prepending timeout command, i.e. timeout 60 docker service scale, because docker service scale was waiting its own timeout even if some containers failed, which could potentially slow down continuous delivery pipelines

References

Docker CLI: docker service scale

jq - command-line JSON processor

GNU Coreutils: timeout command

Upvotes: 1

Denis Pshenov
Denis Pshenov

Reputation: 11347

Yes, there are ways to do it, but it's manual and you'd have to be pretty comfortable with docker cli. Docker does not provide an easy built-in way to verify that docker stack deploy succeeded. There is an open issue about it.

Fortunately for us, community has created a few tools that implement docker's shortcomings in this regard. Some of the most notable ones:

Issuu, authors of sure-deploy, have a very good article describing this issue.

Upvotes: 8

Bret Fisher
Bret Fisher

Reputation: 8626

Typically in CI/CD I see everyone using docker or docker-compose. A container runs the same in docker as it does docker swarm with respects to "does this container work by itself as intended".

That being said, if you still wanted to do integration testing in a multi-tier solution with swarm, you could do various things in automation. Note this would all be done on a single node swarm to make testing easier (docker events doesn't pull node events from all nodes, so tracking a single node is much easier for ci/cd):

  1. Have something monitoring docker events, e.g. docker events -f service=<service-name> to ensure containers aren't dying.
  2. always have healthchecks in your containers. They are the #1 way to ensure your app is healthy (at the container level) and you'll see them succeed or fail in docker events. You can put them in Dockerfiles, service create commands, and stack/compose files. Here's some great examples.
  3. You could attach another container to the same network to test your services remotely 1-by-1 using tasks. with reverse DNS. This will avoid the VIP and let you talk to a specific replica(s).
  4. You might get some stuff out of docker inspect <service-id or task-id>

Upvotes: 1

Related Questions