How should I backup & restore docker named volumes

Question

I'm a little bit confused with the functionnality of named volumes in a docker compose file specifically when it comes to backup/restore my app.

I'm actually testing this dockercompose file :

      version: '2'
      services:
          django:
              build: 
                  context: "{{ build_dir }}/docker/django"
              depends_on:
                  - db
              environment:
                  [...]
              volumes:
                  - code:/data/code
                  - www:/var/www
                  - conf:/data/conf
              networks:
                  - front
                  - db
              expose:
                  - "8080"
              entrypoint: "/init"
          db:
              build:
                  context: "{{ build_dir }}/docker/postgres" 
              environment:
                  [...]
              volumes:
                  - data:/var/lib/postgresql/data
              networks:
                  - db

      volumes:
          data:
          www:
          code:
          conf:

      networks:
          front:
              external:
                  name: "proxy_nw"

As the documentation said, I tried to use named volume instead of data only container. But how am I suppose to backup my data ?

With a data only container I would have done a docker run --rm --volume-from DOC backup_container save which is really easy.

Now I read in this topic that I should use something like docker run --rm --volume data --volume www --volume code --volume conf backup_container save. This is not so simple because I have many applications with different types and names of volumes so it means that my command to save my data would have to be different for each application. It complicate automation process.

Edit: Actually this syntaxe docker run --volume data --volume www container_image my_command is not correct. It needs the mountpoint inside the container, so it would be docker run --volume data:/somewhere --volume www:/somewhereelse container_image my_command. So it's even more complicated to use with a backup container.

So, what are the best practices in this case ? Should I use just one named volume for all my containers ?

Answer 1

I've made a gist with a convenient script for this, I may update it in the future. However here's a copy:

# Function to backup or restore a Docker volume
function docker_volume() {
  (
    ColorOff='\033[0m' # Text Reset
    Yellow='\033[0;33m' # Yellow for important info
    Red='\033[0;31m' # Red for errors
    function infoMessage() {
        echo -e ${Yellow}
        echo $1
        echo -e ${ColorOff}
    }
    function errMessage() {
        echo -e ${Red}
        echo $1
        echo -e ${ColorOff}
    }
    errHandler() {
        # Any steps that must be taken prior to exiting due to error
        errMessage "Something went wrong. Exiting now."
        exit 1
    }
    set -eE # -e throw ERR for any non-zero exit codes, -E as well as in any child functions
    trap 'errHandler' ERR INT # Call errHandler function on ERR (non-zero exit code) or INT (ctrl-c interupt execution)

    local action=$1
    local volume_name=$2
    local file_path=$3

    if [[ -z $action || -z $volume_name || -z $file_path ]]; then
      infoMessage "Command may be used to backup or restore docker volumes to and from tar.gz files"
      infoMessage "Usage: docker_volume <backup|restore> <volume_name> <file_path>"
      infoMessage "Example: docker_volume backup my_volume_name /path/to/backup_file.tar.gz"
      return 1
    fi
    
    # Ensure file_path ends with .tar.gz
    if [[ ! "$file_path" =~ \.tar\.gz$ ]]; then
      infoMessage "File path does not end with .tar.gz, adjusting..."
      file_path="${file_path}.tar.gz"
      infoMessage "File path now set as $file_path"
    fi

    # Check if the volume exists
    if ! docker volume inspect "$volume_name" &>/dev/null; then
      errMessage "Volume '$volume_name' does not exist."
      # Try to list similar volume names
      local similar_volumes=$(docker volume ls --format '{{.Name}}' | grep -i "$volume_name" || true)
      if [[ -n "$similar_volumes" ]]; then
        infoMessage "However, similar volume names were found: $similar_volumes"
      else
        infoMessage "No similar volume names were found. Use \`docker volume ls\` to ensure the volume name is correct."
      fi

      if [ "$action" = "backup" ]; then
        return 1
      elif [ "$action" = "restore" ]; then
        infoMessage "Are you sure you want to proceed with restoring to '$volume_name'? [y/N]"
        read -r response
        if [[ ! "$response" =~ ^[yY]([eE][sS])?$ ]]; then
          errMessage "Restore operation aborted."
          return 1
        fi
      fi
    fi

    case $action in
      backup)
        if [ -f "$file_path" ]; then
          infoMessage "The file $file_path already exists. Overwrite? [y/N]"
          read -r response
          if [[ ! "$response" =~ ^[yY]([eE][sS])?$ ]]; then
            errMessage "Backup operation aborted to prevent overwriting existing file."
            return 1
          fi
        fi
        infoMessage "Starting backup..."
        docker run --rm -v "$volume_name":/volume -v "$(dirname "$file_path")":/backup alpine /bin/sh -c "apk add --no-cache tar > /dev/null && tar czf - -C /volume ./ > /backup/$(basename "$file_path")"
        infoMessage "Volume '$volume_name' has been backed up to '$file_path'"
        ;;
      restore)
        infoMessage "Starting restore..."
        docker run --rm -v "$volume_name":/volume -v "$(dirname "$file_path")":/backup alpine /bin/sh -c "apk add --no-cache tar > /dev/null && tar xzf /backup/$(basename "$file_path") -C /volume"
        infoMessage "Volume '$volume_name' has been restored from '$file_path'"
        ;;
      *)
        errMessage "Invalid action: $action"
        infoMessage "Usage: docker_volume <backup|restore> <volume_name> <file_path>"
        return 1
        ;;
    esac
  )
}

Answer 2

As stated in official documentation there is no out of the box single command to backup/restore volume(s). The one should do it on it's own by mounting a volume to some container and packing the content in some suitable way as given by manuals:

docker run --rm --volumes-from dbstore -v $(pwd):/backup ubuntu tar cvf /backup/backup.tar /dbdata

For plain files stored in a volume this could be enough, but for database it's better to use some dump producing tools. Like mongodump/mongorestore or pg_dump/pg_restore to deal with metadata instead of binary files. As binary files backup has tighter coupling to certain database version.

Some time ago I encountered this issue and developed simple scripts to cover my needs. But with growing of my apps and theirs number I decided to make it common. And recently I open sourced a tool I developed that optionally can upload backups to a Google Drive folder.

pip install cobra-archiver

To backup all the volumes.

cobra backup build

To backup all the volumes but skip-this-one and put the backup in ./backup directory one can use this.

cobra backup build --exclude skip-this-one --backup-dir ./backup

To backup and push to Google Drive.

cobra backup build --push \
    --creds /path/to/google-service-acc-key.json \
    --folder-id google-drive-folder-id \
    --exclude skip-this-volume

Additionally it can backup plain file system directories like this

cobra backup build --dir ./backup/me

To see the cli description on certain command the line is as follows.

$ cobra backup build --help

As well as backing up it allows easy restoration of previously backed up stuff.

cobra backup pull --latest --restore \
    --creds /path/to/google-service-acc-key.json \
    --folder-id google-drive-folder-id

It's also possible to have "dump" database backup automated. To achieve this one can use hooks. Please see the details on github. Real example is available in e2e test sources.

Answer 3

Base on this answer. I made an easy tool here： docker_named_volume_backup

first run command docker volume ls to list the named volume you want to backup.

then for backup

#sudo backup_docker_volume.sh <volumn_name> <tar_file>
sudo bash ./backup_docker_volume.sh codimd_database-data backup1.tar

for restore

#sudo restore_docker_volume.sh <volumn_name> <tar_file>
sudo bash ./restore_docker_volume.sh codimd_database-data backup1.tar

Answer 4

Here is an example of how I backup and restore mongo volume data as docker image.

# let a9c25f42ccca be mongo container id    
docker stop a9c25f42ccca

# create a temp container with volume taken from mongo 
# make a local tar archive inside it
docker run --name temp_backup --volumes-from a9c25f42ccca ubuntu tar cvf /mongo_backup.tar /data/db

docker start a9c25f42ccca

# make an image and remove temp container
docker commit temp_backup my_mongo_backup
docker rm temp_backup

# push image with backup to remote registry if needed
docker push my_mongo_backup

And restoring.

#pull image if needed
docker pull my_mongo_backup

docker stop a9c25f42ccca

# run transient container out from image with backup
# take volume from mongo
# clear any existing data
# restore data from tar arhcive
docker run --rm --volumes-from a9c25f42ccca my_mongo_backup bash -c "rm -rf /data/db/* && tar xvf /mongo_backup.tar -C /data --strip 1"

docker start a9c25f42ccca

Answer 5

Actually it should be done same way as written in official documentation. Data volume container stores it's data in "virtual root", so you should backup with next command:

docker run --rm \ 
  --volume [DOCKER_COMPOSE_PREFIX]_[VOLUME_NAME]:/[TEMPORARY_DIRECTORY_TO_STORE_VOLUME_DATA] \
  --volume $(pwd):/[TEMPORARY_DIRECTORY_TO_STORE_BACKUP_FILE] \
  ubuntu \
  tar cvf /[TEMPORARY_DIRECTORY_TO_STORE_BACKUP_FILE]/[BACKUP_FILENAME].tar /[TEMPORARY_DIRECTORY_TO_STORE_VOLUME_DATA]

where:

• --rm means that the image created for this run command will be cleaned up
• DOCKER_COMPOSE_PREFIX in default is your project directory name
• VOLUME_NAME is the data-volume container name from compose file
• TEMPORARY_DIRECTORY_TO_STORE_VOLUME_DATA is a directory to mount your volume data
• TEMPORARY_DIRECTORY_TO_STORE_BACKUP_FILE is a directory virtually mapped to your current directory, where the backup will be placed
• BACKUP_FILENAME - a name of backup file (you find it in current directory)
• ubuntu - you may change image type to another container with tar :)

Get data back into the volume(restore):

docker run --rm \ 
  --volume [DOCKER_COMPOSE_PREFIX]_[VOLUME_NAME]:/[TEMPORARY_DIRECTORY_STORING_EXTRACTED_BACKUP] \
  --volume $(pwd):/[TEMPORARY_DIRECTORY_TO_STORE_BACKUP_FILE] \
  ubuntu \
  tar xvf /[TEMPORARY_DIRECTORY_TO_STORE_BACKUP_FILE]/[BACKUP_FILENAME].tar -C /[TEMPORARY_DIRECTORY_STORING_EXTRACTED_BACKUP] --strip 1

where:

• TEMPORARY_DIRECTORY_STORING_EXTRACTED_BACKUP is a directory where the extracted files will be copied to (this is linked with the volume and will therefore write to it)
• -C - tell tar where to extract the contents
• --strip 1 - remove leading path elements (e.g. the parent directory if the backup contents are located in a /temp folder or similar)

Answer 6

I finally changed my approach. I parse the volumes of my containers looking for the mountpoints and I backup everything in a separate folder in the tar.

So I'm not using a container to do it but an external script. I don't know if it's a better approach but it works well.