CODEWITHSUNDEEP
bashquoting
will
will

Reputation: 5061

How can prevent bash tokenising string variable expansion?

I don't quite know the term(s) for this part of the bash shell. In one specific, yet critical, case my script falls afoul of this effect:

Objective

Perform this command chain within a script where the awk expression (-e) comes form a variable. This example works when it is a script argument.

echo "test string" | awk  -e { print $0; }

Problem example

On the command line I am seeking to produce output of: "test string", viz.:

$ optE="-e "
$ argE="{ print \$0; }"
$ set -x; echo  "test string" | awk  $optE$argE ; set +x
+ awk -e '{' print '$0;' '}'
+ echo 'test string'
awk: cmd. line:1: {
awk: cmd. line:1:  ^ unexpected newline or end of string
+ set +x

In a way I can see what's happened. Is there a good/best way to not have the $argE variable tokenised after it is expanded?

Typing that same command on the command line works as you know:

$ echo "test string" | awk   -e '{ print $0; }'
test string

Because the expression is enclosed in single quote. I haven't found a way to make that happen using a variable...

$ optE="-e "
$ argE="'{ print \$0; }'"
$ set -x; echo  "test string" | awk  $optE$argE ; set +x
+ echo 'test string'
+ awk -e ''\''{' print '$0;' '}'\'''
awk: cmd. line:1: '{
awk: cmd. line:1: ^ invalid char ''' in expression
+ set +x

Needless to say, I'm on stackoverflow because the things I've tried and read in ohter questions, etc. Don't give desirable result.

Upvotes: 2

Views: 1909

Answers (1)

KamilCuk
KamilCuk

Reputation: 140960

Word splitting expansion is applied after the unquoted variable expansion (${var} or $var) is replaced by its expansion. Word splitting splits the result on (white-)spaces, no matter the quotes. No matter what you put inside the string, if the variable expansion is unquoted, the result will be word split. To affect word splitting, you have to change the way you expand the variable, not it's content (i.e. change $var to "$var").

Is there a good/best way to not have the $argE variable tokenised after it is expanded?

Yes, quote the expansion. Rule of a thumb: never $var always "$var". Also, check your scripts with shellcheck.

In your case, it's simple, just assign the variable the content you want to execute, and quote the expansions:

optE="-e"
argE='{ print $0; }'
echo "test string" | awk "$optE" "$argE" 
                         ^^^^^^^ - variable expansion inside quotes

For more cases, use bash arrays arr=(-e '{ print $0; }'), and properly awk "${arr[@]}" expand them.

Research: https://www.gnu.org/software/bash/manual/html_node/Shell-Operation.html , bash manual shell expansions , https://mywiki.wooledge.org/BashFAQ/050 https://mywiki.wooledge.org/Quotes https://mywiki.wooledge.org/BashPitfalls , When should I wrap quotes around a shell variable? https://github.com/koalaman/shellcheck/wiki/Sc2086 .

Upvotes: 4

Related Questions