How can one avoid expansion errors in functions killing the shell?

Question

I ran into a surprising behavior in one of my shell scripts today. It is demonstrated by the following example:

test.sh

#!/bin/bash

do_it() {
  shopt -s failglob
  {
    rm killme.*
    echo "and then ..."
  } 2>/dev/null || echo "glob error"

  echo "life goes on ..."
}

do_it || echo "function failed"

The idea in the original script was that I wanted to allow glob expansion errors to occur for a particular command so as to avoid executing that one command when there were no arguments, yet detect that error and take alternative action. My expectation was that when killme.* did not match anything, executing the above script via

./test.sh || echo "script failed"

would emit

glob error
life goes on ...

or maybe

function failed

. It didn't (with Bash 4.2.46). Instead it printed

script failed

. While troubleshooting the issue, I discovered something even more curious: if I simplify the script further by eliminating the function, the behavior changes. That is, consider this alternative script:

test2.sh

#!/bin/bash

shopt -s failglob

{
  rm killme.*
  echo "and then ..."
} 2>/dev/null || echo "glob error"

echo "life goes on ..."

If I run that via

./test2.sh || echo "script failed"

, it prints

life goes on ...

There seem to be some other weird variations when a function like the one in the first script is called in a loop, but I haven't fully characterized that.

Questions:

1. Is this documented behavior? My examination of the Bash manual has been unavailing. It specifies that an "expansion error" occurs, and it seems natural that that's a shell error, not an error in the command, but if there's anything by which I should be able to predict the details of the observed results then I've missed it.

2. I can solve the issue by running the expansion in a subshell, but is there any lighter weight workaround? I guess I could perform the expansion in advance, with failglob unset, and test the result, but that's messy and contains a race condition.

Answer 1

1. Is this documented behavior? My examination of the Bash manual has been unavailing. It specifies that an "expansion error" occurs, and it seems natural that that's a shell error, not an error in the command, but if there's anything by which I should be able to predict the details of the observed results then I've missed it.

Neither the current version of the Bash manual nor the manual for the version on which I discovered the problem seems to document the behavior that should be expected when pathname expansion results in an expansion error. Some versions of the manual do document behavior when an error occurs during parameter expansion, and apparently that varies between some Bash versions and depending on whether Bash is running in POSIX mode.

POSIX itself, however, does not distinguish between different varieties of expansion error. It specifies that (all) expansion errors in non-interactive shells cause the shell to terminate with a diagnostic. This is the behavior exhibited by my test.sh, but it conflicts with the behavior exhibited by my test2.sh.

Inasmuch as Bash does not claim to be fully POSIX-conformant, especially when not running in POSIX-compatibility mode, the discrepancy with POSIX cannot be considered a bug, but the surprising and undocumented inconsistency between the behavior of the two scripts certainly seems buggy to me, so I have filed an issue about it.

2. I can solve the issue by running the expansion in a subshell, but is there any lighter weight workaround? I guess I could perform the expansion in advance, with failglob unset, and test the result, but that's messy and contains a race condition.

Inasmuch as the POSIX shell is expected to terminate on expansion errors, and Bash does this in at least some contexts, the only safe way to enable recovery from expansion errors seems to be to cause the error to occur in a subshell.

In my particular case, I want to ensure that the command does not run with zero arguments or with an unexpanded glob as an argument, as it may do the wrong thing in these cases. It is OK, though, if the command runs with arguments designating files that disappear after expansion but before the command tries to operate on the arguments. The simplest and most reliable way to accomplish seems to be to just run the command in a subshell in which the failglob option is set.

It would also be possible to pre-expand the glob with failglob unset, test whether it is empty, and then use the pre-expanded result instead of expanding the glob again. That seems like overkill for my purposes, so I have opted to go with the subshell.

Update

As @eurythmia pointed out in his answer, Bash's behavior is even stranger than I thought. In test1.sh, the expansion error does not cause the whole script to terminate; rather, it causes the pipeline containing the do_it call to fail immediately, altogether bypassing any consideration of the success or failure of the function call command itself, and for that reason not executing the echo command. For all intents and purposes, the do_it call itself neither succeeds nor fails, which is highly peculiar at best.

That does not, however, change the conclusions presented in this answer: the behavior is definitely not documented for Bash, and it is inconsistent with POSIX. The simplest safe alternative is to isolate any use of the failglob option inside a subshell, but if that's impractical for some reason then there are workarounds.

Answer 2

As the context for my above comment ... you can use a variable with shell globbing to check if a file with the pattern exists, and remove them all if they do, else print out the error message. This leaves you without depending on an error state to trigger the "this didn't work" message.

shopt -s nullglob
a=(killme.*)
if [[ -n $a ]]; then
 rm killme.* > /dev/null 2>&1
 echo "life goes on ...
else
  echo "glob error"
fi 

... I can't (at present) offer any insight into why the function is failing, other than the rumination that perhaps function calls are executed as subshells.

Edit: I found this gem in bash's subst.c ... it looks like we jump up to the top shell context, discarding all current context, and set the failure code:

  else if (fail_glob_expansion != 0)
    {
      last_command_exit_value = EXECUTION_FAILURE;
      report_error (_("no match: %s"), tlist->word->word);
      exp_jump_to_top_level (DISCARD);
    }

... in this case, I suspect that bash is parsing doit || echo "function failed" as a single command, causing the whole bit to fail. Since bash returns the exit code of the last command in a script, that explains why you're seeing your script 'fail' (i.e. ./my_script.sh || echo "script failed" is printing "script failed").

You can see that if you add echo "exit_code: $?" as the last line of your script it will print a non-zero code (i.e. failure), but your script will return a success code:

[eurythmia@localhost ~]$ cat ./test_script.sh 
#!/bin/bash

do_it() {
  shopt -s failglob
  {
    rm killme.*
    echo "and then ..."
  } >/dev/null 2>&1 || echo "glob error"

  echo "life goes on ..."
} 

do_it || echo "function failed"
echo "exit code: $?"
[eurythmia@localhost ~]$ ./test_script.sh || echo "I failed"
exit code: 1
[eurythmia@localhost ~]$ echo $?
0
[eurythmia@localhost ~]$ 

I guess this all comes down to how bash parses, and what it considers to be a 'command' (something that I'm going to look further into, for my own edification). In the meantime, I would not depend on shopt -s failglob from within a function. Viable alternatives include using shopt -s failglob from the root level of the script, or sticking to the standard test operators (which are implemented as bash builtins) when working inside functions.