CODEWITHSUNDEEP
gnu-parallel
jon doe
jon doe

Reputation: 544

Write output to stdout as well as results in GNU parallel

I'm using GNU parallel to run multiple python scripts in parallel and each one can take a few minutes to complete. The scripts print out their progress to stdout.

An example script.py

print('doing something...')
do_something()
print('doing another thing...')
do_another_thing()

My parallel command which only saves output to a results directory tree

parallel --lb --results results < parallel-commands.txt

The content of parallel-commands.txt

python3 script.py --once
python3 script.py --twice

I want to print the output when it's ready, and I looked online and found the --lb option for parallel, which helped me, but I also want parallel to write the output to a file as well as to stdout, kind of like what tee does. Is there any way to achieve that?

Upvotes: 1

Views: 205

Answers (1)

Ole Tange
Ole Tange

Reputation: 33740

Currently GNU Parallel only supports a single output stream. In other words you cannot get output in both stdout and files.

So you need to do it manually like this:

#!/bin/bash

doit() {
    echo $1 start
    sleep $1
    echo $1 end
}
export -f doit

parallel --lb 'doit {} | tee {#}.out' ::: 3 2 1

It is, however, a feature that is under consideration, so it is good to know that there will be atleast 1 user using it :)

Upvotes: 1

Related Questions