CODEWITHSUNDEEP
linuxbashgnu-parallel
Physicus
Physicus

Reputation: 71

GNU parallel: execute one command parallel for all files in a folder

I am trying to parallelize particle simulations with different parameters to save some time. Therefore I wanted to use GNUparallel to run a bash script for the different parameters. The script reads a file and then performs the simulation eg :

$bash script <<< input file

However:-

$cd ~/parameter_files ls | parallel bash script <<< {}

does not work at all. I am a total newbie to Linux and GNUparallel, so hopefully someone can help.

Upvotes: 7

Views: 6710

Answers (2)

etopylight
etopylight

Reputation: 1319

You're almost right, use double quotes around the command

ls | parallel "bash script <<< {}"

Otherwise the here string <<< would feed the input into the parallel command rather than each individual bash script commands

I find this use case to be quite unusual however, since it means that basically your script is reading the filename string. If you just want to pass the files as arguments to the script, you can use

ls | parallel bash script

or if you want to pass the content of the files using standard input

ls | parallel "bash script < {}"

Upvotes: 6

anubhava
anubhava

Reputation: 785216

You can use parallel like this on each file of ~/parameter_files directory:

find ~/parameter_files -maxdepth 1 -type f -print0 | parallel -0 bash ../script
  • find command finds all the files inside ~/parameter_files directory and returns them NUL terminated
  • parallel -0 bash ... feeds each filename to your shell script and executes it.

Upvotes: 1

Related Questions