CODEWITHSUNDEEP
bashxargs
Ian Fiddes
Ian Fiddes

Reputation: 3001

Saving arguments passed from xargs to bash as a variable for processing

I am trying to run many commands in parallel, and need to do some string manipulation on the input first. How can I make the below example work?

find . -mindepth 1 -type d | xargs -n 1 -P 20 -i sh -c "v={}; echo $v"

When I use this, $v is null. Why is it not being saved as the value of {}?

Upvotes: 10

Views: 19351

Answers (2)

Ole Tange
Ole Tange

Reputation: 33685

With GNU Parallel you would do:

find . -mindepth 1 -type d | parallel -P 20 'v={}; echo "$v"'

Upvotes: 1

Mike Holt
Mike Holt

Reputation: 4612

The parent shell is expanding $v before the string gets passed to xargs.

Suppose your find command finds a subdirectory named ./stuff.

First, the parent bash shell (the one you typed the find command into) will expand $v, because the string is in double quotes. You currently have no value set for variable v, so it expands to an empty string.

Next, the arguments get passed to xargs, which will see this: v={}; echo

Then, xargs will read ./stuff from the pipe, and replace {} with ./stuff

Finally, the sh command is executed by xargs, and sh will see this: v=./stuff; echo

To fix this, you need to either escape the $ so that the parent shell doesn't expand it, or use single quotes to avoid variable expansion. You should also probably quote the strings so that any directory names with spaces in them don't cause problems with the final sh command:

find . -mindepth 1 -type d | xargs -n 1 -P 20 -i sh -c "v=\"{}\"; echo \"\$v\""

OR

find . -mindepth 1 -type d | xargs -n 1 -P 20 -i sh -c 'v="{}"; echo "$v"'

With either command, the final sh process will see: v="./stuff"; echo "$v"

By the way, one way to see for yourself that this is indeed what is happening would be to set a value for v in the parent shell, then run your original command. The shell will expand $v to whatever value you set, and you will see that value repeated for every directory found by find.

$ v=foobar
$ find . -mindepth 1 -type d | xargs -n 1 -P 20 -i sh -c "v={}; echo $v"
foobar
foobar
foobar
foobar
foobar
foobar
foobar
foobar
foobar
foobar
foobar
...

Upvotes: 14

Related Questions