CODEWITHSUNDEEP
dockershelldocker-composedata-science
TApplis
TApplis

Reputation: 97

How do I run multiple docker-compose files AFTER each other?

I want to perform a set of experiments, where every experiment is described as a docker-compose. The structure is something like follows:

Dataset
    A
       1 
          docker-compose-1.yml
          docker-compose-2.yml
       2 
          docker-compose-1.yml
          ...
    B
       1 
          docker-compose-1.yml
          ...

All the composes are well-behaved, that is I can/could run them manually after each other and they behave as expected.

I tried two attempts, the first is a shell file as the following:

#!/bin/bash

target_dir=../dataset

localizedCompose() {
    f=$1;
    ff=`basename ${f}`
    d=`dirname ${f}`

    l=`pwd`
    cd $d
    echo "Docker-Composing ${d}/${ff}"
    docker-compose -f $ff up
    echo "Docker-Compose finished ${d}/${ff}"
    docker-compose -f $ff down 
    echo "Docker-Compose cleaned-up"
    cd $l
}

export -f localizedCompose;

find $target_dir -name 'docker-compose*.yaml' -print0 | sort -zn | xargs -I {} bash -c 'localizedCompose "$@"' _ {}

The | sort -zn is to run the files in order, which is not super-duper important but would be nice.

Which ends after the first compose. Similarly, I tried to run it "just with find" such as

find ./dataset -type f -name docker-compose*.yml -exec bash -c "docker-compose -f {$1} up ; docker-compose -f {$1} down" {} \;

This (suprisingly) also runs the first compose it finds correctly but also exits after that.

As the experiments are resource expensive and need to be timed, they really have to run after each other, I cannot merge them into one big compose.

Upvotes: 1

Views: 811

Answers (1)

David Maze
David Maze

Reputation: 158647

You're not passing a -0 argument to xargs. Without this, it interprets its stdin as a sequence of space-separated strings; since there are no spaces, it runs its command only once; but a typical convention for C programs is to interpret a null byte as end-of-string, so you'll only get the first filename out in practice.

find ... -print0 | xargs -0 ...

In practice I'd try to simplify this, probably using a for loop

for f in $(find "$target_dir" -name 'docker-compose*.yaml' -print | sort -n)
do
  localizedCompose "$f"
done

This could hit some hiccups if any of the file or directory names have spaces in them. If you're okay using bash-specific syntax, you should be able to avoid this problem:

shopt -o failglob globstar
for f in **/docker-compose*.yaml; do # automatically sorted
  localizedCompose "$f"
done

Within the function, Compose generally ignores the working directory, and considers all paths relative to the location of the (first) -f option. You can trim this function down to

localizedCompose() {
  echo "docker-compose up $1"
  docker-compose -f "$1" up
  echo "docker-compose down $1"
  docker-compose -f "$1" down
  echo "done with $1"  
}

Upvotes: 1

Related Questions