CODEWITHSUNDEEP
linuxbashshellparallel-processing
user3595106
user3595106

Reputation: 1

Run block of commands parallelly & save output to a file sequentially

I'm looking for a Linux Shell script where I can Run a block of commands in parallel & save output to a file sequentially.

For example:

for each line in file.txt
{
    Execute A
    Execute B
    Execute C > Save output to output.txt
}

How do we make sure the output.txt has sequential output & the block of commands can be run in parallel.

Upvotes: 0

Views: 173

Answers (1)

Renaud Pacalet
Renaud Pacalet

Reputation: 28955

The wait bash builtin waits for all background jobs. So redirecting their output to a temporary file, waiting until they end, and concatenating the temporary files should be about what you want. Example where we replace Execute A by ( echo "A"; sleep 5 )`:

if tA=$(mktemp); then ( echo "A"; sleep 5 ) > "$tA" & fi
if tB=$(mktemp); then ( echo "B"; sleep 5 ) > "$tB" & fi
if tC=$(mktemp); then ( echo "C"; sleep 5 ) > "$tC" & fi
wait
cat "$tA" "$tB" "$tC" > output.txt
rm -f "$tA" "$tB" "$tC"

As noted by Léa Gris, when creating temporary files with mktemp it is always better to ensure that they are properly deleted even if the process is terminated before it explicitly deletes them. So the following is cleaner:

unset tA tB tC
trap 'rm -f -- "$tA" "$tB" "$tC"' EXIT TERM
if tA=$(mktemp); then ( echo "A"; sleep 5 ) > "$tA" & fi
if tB=$(mktemp); then ( echo "B"; sleep 5 ) > "$tB" & fi
if tC=$(mktemp); then ( echo "C"; sleep 5 ) > "$tC" & fi
wait
cat "$tA" "$tB" "$tC" > output.txt

Upvotes: 4

Related Questions