CODEWITHSUNDEEP
linuxshellcommand-lineparallel-processingxargs
Charlie Hemlock
Charlie Hemlock

Reputation: 11

Can xargs be used to run several arbitrary commands in parallel?

I'd like to be able to provide a long list of arbitrary/different commands (varying binary/executable and arguments) and have xargs run those commands in parallel (xargs -P).

I can use xargs -P fine when only varying arguments. It's when I want to vary the executable and arguments that I'm having difficulty.

Example: command-list.txt

% cat command-list.txt
binary_1 arg_A arg_B arg_C 
binary_2 arg_D arg_E
.... <lines deleted for brevity>
binary_100 arg_AAA arg_BBB

% xargs -a command-list.txt -P 4 -L 1
 ** I know the above command will only echo my command-list.txt **

I am aware of GNU parallel but can only use xargs for now. I also can't just background all the commands since there could be too many for the host to handle at once.

Solution is probably staring me in the face. Thanks in advance!

Upvotes: 1

Views: 2510

Answers (1)

inthecrossfire
inthecrossfire

Reputation: 153

If you don't have access to parallel, one solution is just to use sh with your command as the parameter.

For example:

xargs -a command-list.txt -P 4 -I COMMAND sh -c "COMMAND"

The -c for sh basically just executes the string given (instead of looking for a file). The man page explanation is:

-c string   If  the  -c  option  is  present, then commands are read from
            string.  If there are arguments after the  string,  they  are
            assigned to the positional parameters, starting with $0.

And the -I for xargs tells it to run one command at a time (like -L 1) and to search and replace the parameter (COMMAND in this case) with the current line being processed by xargs. Man page info is below:

-I replace-str
            Replace occurrences of replace-str in the initial-arguments with
            names read from standard input.  Also, unquoted  blanks  do  not
            terminate  input  items;  instead  the  separator is the newline
            character.  Implies -x and -L 1.

sh seems to be very forgiving with commands containing quotations marks (") so you don't appear to need to regexp them into escaped quotations.

Upvotes: 3

Related Questions