CODEWITHSUNDEEP
arraysbashshellifs
Norswap
Norswap

Reputation: 12222

Surprising array expansion behaviour

I've been surprised with the line marked (!!) in the following example:

log1 () { echo $@; }
log2 () { echo "$@"; }

X=(a b)
IFS='|'

echo ${X[@]}   # prints a b
echo "${X[@]}" # prints a b
echo ${X[*]}   # prints a b
echo "${X[*]}" # prints a|b
echo "---"
log1 ${X[@]}   # prints a b
log1 "${X[@]}" # prints a b
log1 ${X[*]}   # prints a b
log1 "${X[*]}" # prints a b (!!)
echo "---"
log2 ${X[@]}   # prints a b
log2 "${X[@]}" # prints a b
log2 ${X[*]}   # prints a b
log2 "${X[*]}" # prints a|b

Here is my understanding of the behavior:

This seems to be confirmed by the bash manual.

In the line log1 "${X[*]}", I therefore expect the quoted expression to expand to "a|b", then to be passed to the log1 function. The function has a single string parameter which it displays. Why does something else happen?

It'd be cool if your answers were backed by manual/standard references!

Upvotes: 12

Views: 575

Answers (3)

Etan Reisner
Etan Reisner

Reputation: 81052

In the POSIX spec section on [Special Parameters[(http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_05_02) we find.

@

Expands to the positional parameters, starting from one. When the expansion occurs within double-quotes, and where field splitting (see Field Splitting) is performed, each positional parameter shall expand as a separate field, with the provision that the expansion of the first parameter shall still be joined with the beginning part of the original word (assuming that the expanded parameter was embedded within a word), and the expansion of the last parameter shall still be joined with the last part of the original word. If there are no positional parameters, the expansion of '@' shall generate zero fields, even when '@' is double-quoted.

*

Expands to the positional parameters, starting from one. When the expansion occurs within a double-quoted string (see Double-Quotes), it shall expand to a single field with the value of each parameter separated by the first character of the IFS variable, or by a if IFS is unset. If IFS is set to a null string, this is not equivalent to unsetting it; its first character does not exist, so the parameter values are concatenated.

So starting with the quoted variants (they are simpler):

We see that the * expansion "expand[s] to a single field with the value of each parameter separated by the first character of the IFS variable". This is why you get a|b from echo "${X[*]" and log2 "${X[*]}".

We also see that the @ expansion expands such that "each positional parameter shall expand as a separate field". This is why you get a b from echo "${X[@]}" and log2 "${X[@]}".

Did you see that note about field splitting in the spec text? "where field splitting (see Field Splitting) is performed"? That's the key to the mystery here.

Outside of quotes the behavior of the expansions is the same. The difference is what happens after that. Specifically, field/word splitting.

The simplest way to show the problem is to run your code with set -x enabled.

Which gets you this:

+ X=(a b)
+ IFS='|'
+ echo a b
a b
+ echo a b
a b
+ echo a b
a b
+ echo 'a|b'
a|b
+ echo ---
---
+ log1 a b
+ echo a b
a b
+ log1 a b
+ echo a b
a b
+ log1 a b
+ echo a b
a b
+ log1 'a|b'
+ echo a b
a b
+ echo ---
---
+ log2 a b
+ echo a b
a b
+ log2 a b
+ echo a b
a b
+ log2 a b
+ echo a b
a b
+ log2 'a|b'
+ echo 'a|b'
a|b

The thing to notice here is that by the time log1 is called in all but the final case the | is already gone.

The reason it is already gone is because without quotes the results of the variable expansion (in this case the * expansion) are field/word split. And since IFS is used both to combine the fields being expanded and then to split them again the | gets swallowed by field splitting.

And to finish the explanation (for the case actually in question), the reason this fails for log1 even with the quoted version of the expansion in the call (i.e. log1 "${X[*]}" which expands to log1 "a|b" correctly) is because log1 itself does not use a quoted expansion of @ so the expansion of @ in the function is itself word-split (as can be seen by echo a b in that log1 case as well as all the other log1 cases).

Upvotes: 3

chepner
chepner

Reputation: 532313

IFS is used not just to join the elements of ${X[*]}, but also to split the unquoted expansion $@. For log1 "${X[*]}", the following happens:

  1. "${X[*]}" expands to a|b as expected, so $1 is set to a|b inside log1.
  2. When $@ (unquoted) is expanded, the resulting string is a|b.
  3. The unquoted expansion undergoes word-splitting with | as the delimiter (due to the global value of IFS), so that echo receives two arguments, a and b.

Upvotes: 7

choroba
choroba

Reputation: 242343

That's because $IFS is set to |:

(X='a|b' ; IFS='|' ; echo $X)

Output:

a b

man bash says:

IFS The Internal Field Separator that is used for word splitting after expansion ...

Upvotes: 5

Related Questions