CODEWITHSUNDEEP
dockerdocker-compose
Marcin Kunert
Marcin Kunert

Reputation: 6295

How to achieve a rolling update with docker-compose?

I have a following setup in docker compose

Frontend can be updated relatively fast using

docker-compose up -d --no-deps frontend

Unfortunately backend takes about 1 minute to start.

Is there an easy way to achieve lower downtime without having to change the current setup too much? I like how simple it is right now.

I would imagine something like this:

  1. Start a new instance of backend
  2. Wait till it starts (it could be per timer or a healthtest)
  3. Close the previously running instance

Upvotes: 35

Views: 26497

Answers (3)

majidkabir
majidkabir

Reputation: 151

I have created a repository that contains script for achieving the rolling update with docker-compose. It is based on the answers in this page. The steps are as below:

  1. Updating the images in docker-compose file for the services that we want to update.
  2. Scaling up the updated services
  3. Waiting for the services to pass the starting state.
  4. Checking the status of services, if there is an unhealthy container, stop that container, and if all of them are healthy just stop the oldest one.

The script is working for 1 replica, but it can be modified to support services with more replica.

I will put the main 2 scripts here but you can find them in the repository here

Script for checking the service containers and stoping the unhealthy or oldes one:

#!/bin/bash

 # Function to get container IDs for a service in a specific state
 get_containers_in_state() {
     local service=$1
     local state=$2
     docker ps --filter "label=com.docker.compose.service=${service}" --filter "health=${state}" -q
 }

 # Function to get all container IDs for a service sorted by creation time (oldest first)
 get_all_containers_sorted_by_age() {
     local service=$1
     docker ps --filter "label=com.docker.compose.service=${service}" -q | xargs docker inspect --format '{{.Created}} {{.Id}}' | sort | awk '{print $2}'
}

stop_and_remove_container() {
    local container=$1
    docker stop "${container}"
    docker rm "${container}"
    docker exec nginx nginx -s reload
}

 # Main script
 service_name=$1

 if [ -z "${service_name}" ]; then
     echo "Usage: $0 <service_name>"
     exit 1
 fi

 # Wait until no containers are in the "starting" state
 timeout=5  # Set the timeout duration in seconds
 elapsed=0
 echo "Waiting for containers of service '${service_name}' to be in a stable state..."
 while true; do
     starting_containers=$(get_containers_in_state "${service_name}" "starting")
     if [ -z "${starting_containers}" ]; then
         break
     fi

     if [ ${elapsed} -ge ${timeout} ]; then
         echo "Timeout reached. Stopping the container."
         for container in ${starting_containers}; do
             stop_and_remove_container "${container}"
         done
         exit 0
     fi

     sleep 1
     elapsed=$((elapsed + 1))
 done

 # Check for unhealthy containers
 unhealthy_containers=$(get_containers_in_state "${service_name}" "unhealthy")

 if [ -n "${unhealthy_containers}" ]; then
     echo "Stopping unhealthy container(s): ${unhealthy_containers}"
     for container in ${unhealthy_containers}; do
         stop_and_remove_container "${container}"
     done
 else
     # Get the oldest container for the service
     all_containers=$(get_all_containers_sorted_by_age "${service_name}")
     container_count=$(echo "${all_containers}" | wc -l)

     if [ "${container_count}" -gt 1 ]; then
         oldest_container=$(echo "${all_containers}" | head -n 1)
         echo "Stopping the oldest container: ${oldest_container}"
         docker exec nginx nginx -s reload
         stop_and_remove_container "${oldest_container}"
     else
         echo "No containers found for service '${service_name}'."
     fi
 fi

The script for scaling up the services and calling the first script for the services that we want to update

#!/bin/bash

docker compose up -d --scale service1=2 --scale service2=2 --no-recreate

./check_status.sh service1 & ./check_status.sh service2

Upvotes: 1

Marcin Kunert
Marcin Kunert

Reputation: 6295

Here is the script I've ended up using:

PREVIOUS_CONTAINER=$(docker ps --format "table {{.ID}}  {{.Names}}  {{.CreatedAt}}" | grep backend | awk -F  "  " '{print $1}')
docker-compose up -d --no-deps --scale backend=2 --no-recreate backend
sleep 100
docker kill -s SIGTERM $PREVIOUS_CONTAINER
sleep 1
docker rm -f $PREVIOUS_CONTAINER
docker-compose up -d --no-deps --scale backend=1 --no-recreate backend
docker-compose stop http-nginx
docker-compose up -d --no-deps --build http-nginx

Upvotes: 19

Siyu
Siyu

Reputation: 12139

Swarm is the right solution to go, but this is still painfully doable with docker-compose.

First, ensure your proxy can do service discovery. You can't use container_name (as you can't use it in swarm) because your will increase the number of container of the same service. Proxy like traefik or nginx-proxy uses labels to do this.

Then, docker-compose up -d --scale backend=2 --no-recreate this creates a new container with the new image without touching the running one.

After it's up and running, docker kill old_container, then docker-compose up -d --scale backend=1 --no-recreate just to reset the number.

EDIT 1

docker kill old_container should be docker rm -f old_container

EDIT 2

how to handle even and not even runs

We want to always kill the oldest containers

docker rm -f $(docker ps --format "table {{.ID}}  {{.Names}}  {{.CreatedAt}}" | grep backend | (read -r; printf "%s\n" "$REPLY"; sort -k 3 ) | awk -F  "  " '{print $1}' | head -1)

Upvotes: 21

Related Questions