String concatenated as an argument for a command

Question

In bash shell, I would like to list file names under $folder (variable name in my code) with full path.

However, if I code like this,

dir $folder"*"

It outputs errors

"dir: cannot access <folder_name> : No such file or directory"

It works if I change the code like this,

search_str=$folder"*"
dir $search_str 

It seems that the concatenated string to dir command is not what I intended and if I store the string in a variable and pass it only to dir command it works.

Anyone knows the reason & how not to use search_str?

Answer 1

While @mklement0's answer explains the cause of the error, you may also look for possible other solutions. First let me borrow from the other answer the below line

Pattern (globbing) metacharacters such as * must be used unquoted in order to be recognized as such.

Case 1: If you wish to list(not parse, just list) the files(files only) recursively inside folder,
use [ globstar ].

shopt -s globstar
folder="yourfoldername**"
ls -p $folder | grep -v '/$'   #mind that '$folder' is not inside double quotes
# '-p' adds trailing slash to directories and we neglect those results using grep invert match
# Here I inadvertantly parsed the `ls` output, see the link in case 2 for why  you shouldn't do that.

Case 2: If you wish to parse(not just list) the files(files only) recursively inside a folder, use find as ls output cannot be parsed.^{[Check this]}

folder="yourfoldername" #no trailing **
find "$folder" -type f -print0 | while read -r -d '' fname
do
#do something with "$fname"
done

Answer 2

^{Note: I'm using standard Unix utility ls rather than GNU-specific utility dir (thanks, hek2mgl) in the examples below, because the issue relates to generic shell behavior and is not limited to platforms with GNU utilities.}

tl;dr

ls "$folder"* # double-quote the var. reference, use * unquoted

---

Pattern (globbing) metacharacters such as * must be used unquoted in order to be recognized as such.

Thus, ls $folder"*" will invariably not work as intended, because you (double-)quoted *.

By contrast, variable assignment search_str=$folder"*" causes the * to be appended to the value of $folder (as a literal, because it is double-quoted, but ultimately without quotes, because the shell removes the double quotes in the process of the string concatenation due to their having syntactic meaning rather than being literals; this removal is fittingly called quote removal).

It is the unquoted use of the variable in command ls $search_str that then makes the * effective as a wildcard (pattern metacharacter).

However, this unquoted use of a variable is inherently not robust, because the value of $search_str is as a whole subject not only to globbing, but also to word-splitting, which means that the command would break if the original $folder variable contained a path with embedded spaces.

Therefore it is better to append the * on demand, when globbing is actually needed:

ls "$folder"*

Note how * is unquoted - to make it effective as a pattern metacharacter - whereas "$folder" is intentionally double-quoted - to ensure it is used as-is (word-splitting and globbing is not applied to its value).

Answer 3

suppose $folder is tmp then dir $folder"*" is equivalent to dir tmp*

while in case of below

search_str=$folder"*"
dir $search_str 

$search_str is equivalent to dir temp

when you use double quotes, it needs an extra step to expand. See more on Variable expansion in shell