CODEWITHSUNDEEP
bashgnu-parallel
LookAheadAtYourTypes
LookAheadAtYourTypes

Reputation: 1689

how to fix incorrect gnu parallel substring extraction

I've tried to use substring extraction in bash script mixed with gnu parallel. But below code (simplified from much more complicated case) produces wrong results.

#!/bin/bash                                                        

function foo(){                                                    
  echo "${1:0:1} ${1:1:1}" # substring extraction                  
}                                                                  

function bar(){                                                    
  IFS=', ' read -r -a array <<< "${1}" # string to array conversion
  echo "${array[0]} ${array[1]}"                                   
}                                                                  

export -f foo                                                      
export -f bar                                                      

values=( '12' '34' )                                               

parallel echo $(foo {} ) ::: "${values[@]}"                        
# produces wrong output...                                         
# {} 12                                                            
# {} 34                                                            

parallel echo $(bar {} ) ::: "${values[@]}"                        
# produces wrong output...                                         
# 12                                                               
# 34

Could you provide me some hint how can I convince gnu parallel to assume that a variable inside function exists and are not ony brackets.

Upvotes: 1

Views: 323

Answers (1)

Mike Holt
Mike Holt

Reputation: 4612

I think what you're missing is that bash will do the process substitution $(foo {} ) before it passes arguments to parallel. You can see this if you replace parallel with printf "%s\n":

printf "%s\n" echo $(foo {} ) ::: "${values[@]}"
echo
{
}
:::
12
34

Which means your command is equivalent to this:

parallel echo { } ::: 12 34

And hence why it prints { } 12 and { } 34. There is no {} here for parallel to replace, since foo has split it into two separate args, { and }. So just like xargs does when there's no {}, parallel just tacks on the args to the end of the command, yielding the commands:

echo { } 12
echo { } 34

To delay the process substitution, you need to wrap it in single quotes:

parallel echo '$(foo {} )' ::: "${values[@]}"

However, this leads to another problem, because the process spawned by parallel won't recognize function foo. But you can solve that with export -f:

export -f foo
parallel echo '$(foo {} )' ::: "${values[@]}"
1 2
3 4

Likewise for your bar example.

Edit: Your bar example still prints the same as it did before, but for a different reason. You're trying to read the first argument to bar into array, with IFS=', ', but your input doesn't contain any commas (or spaces), so you get an array of one element each time, and array[1] expands to nothing.

But if you do this instead, it works (or at least I think it does - I'm not sure what your expected output was for this example):

values=( "1,2" "3,4" )
parallel echo '$(bar {} )' ::: "${values[@]}"
1 2
3 4

Upvotes: 2

Related Questions