CODEWITHSUNDEEP
bashwaitchild-processgnu-parallel
alex
alex

Reputation: 11410

Bash wait does not work

Part of my code:

for i in "${r_arr[@]}"
do
    ${parallel_dir}parallel -j${cap} --semaphore --semaphorename a2_bootstrap_semaphore_${rnd} a2_exec ${i} ${temp_zmat_folder_path}${temp_zmat_folder}/ ${outs_folder}
done

wait
elapsed_time=$(($SECONDS - $start_time))

rm -rf "${temp_zmat_folder_path}${temp_zmat_folder}"
echo "Cleanup done."
echo "`date +%d-%m-%y` `date +%T`: All processing finished in ${elapsed_time} seconds!"

As you can see I'm starting a number of tasks using GNU Parallel. The problem is that cleanup (rm) is executed before last of the parallel tasks is even started. How to prevent such behaviour?

Upvotes: 4

Views: 806

Answers (2)

Sam Daniel
Sam Daniel

Reputation: 1902

As per their example, you should use this command to wait for all the started jobs to complete

parallel --semaphore --wait

Example (From man page): The command sem is an alias for parallel --semaphore.

for i in *.log ; do
    echo $i
    sem -j+0 gzip $i ";" echo done
done
sem --wait

Upvotes: 4

Ole Tange
Ole Tange

Reputation: 33748

Sam Daniels answer is entirely correct (except you need to wait for the --semaphorename that you use), but sem is really slow. Consider using a function instead:

doit() {
  i="$1"
  a2_exec ${i} ${temp_zmat_folder_path}${temp_zmat_folder}/ ${outs_folder}
}
export -f doit
export temp_zmat_folder_path
export temp_zmat_folder
export outs_folder
${parallel_dir}parallel -j${cap} doit ::: "${r_arr[@]}"

Upvotes: 0

Related Questions