CODEWITHSUNDEEP
gnu-parallel
JB_User
JB_User

Reputation: 3267

How to use GNU Parallel to run a list of command in a file?

I would like to use GNU Parallel to run N commands on P cores. I put each command in a file: ListOfCommand. Each line in that file corresponds to a different command and each line is the complete command that can be run 'as is' at the command-line. There are many more commands than cores. So I would like GNU Parallel to run only P commands at a time, and then whenever a command finishes running, the next command in the file should begin (dynamic scheduling).

It seems like this should be doable with GNU Parallel, but I'm having a hard time digesting the man page. Does anyone know how to do this?

[EDIT] After more digging, this seems to work:

parallel -P 4 '{}' < ListOfCommands

Upvotes: 2

Views: 742

Answers (1)

Mark Setchell
Mark Setchell

Reputation: 207798

As regards supplying the list of commands, you should be able to use any of the following equivalently:

parallel -j4 < ListOfCommands
parallel -j4 -a ListOfCommands
parallel -j4 :::: ListOfCommands

As regards what is run for each command in the list, you can use {} to identify parameters, but if you specify no command to be run, the line from your file is taken as the command:

So, if you make your ListOfCommands like this:

echo A
echo B
echo C

You can run the following and the command to run for each job is the echo X line from your commands:

parallel -j4 :::: ListOfCommands
A
B
C

If you just want to put the parameters for the command to run in a file, you write ListOfParameters like this:

A
B
C

then put the echo command on the GNU Parallel command line itself as follows and use {}:

parallel -j4 echo {} :::: ListOfParameters
A
B
C

Upvotes: 2

Related Questions