CODEWITHSUNDEEP
bashshell
Red Cricket
Red Cricket

Reputation: 10470

grep command in my script return empty results but works from command line

I am trying to write a bash script that is suppose to find the file that has a specific string in it. The script calls another script that returns strings of the format:

url=title

title is the string I am looking for. title can have values that look like this for example: 'A Soldier Of The Legion'. I am trying to find the file in the /tmp/audiobooksdirectory that contains the title, 'A Soldier Of The Legion'. All the files in /tmp/audiobooks have names that end with AB.yaml.

Here is the script: 

#!/bin/sh

get_pairs='/home/me/util/scripts/get-pairs.sh'
SAVEIFS=$IFS
IFS=$(echo -en "\n\b")
for i in `$get_pairs`
do
        echo "pair $i"
        url=`echo $i | cut -d= -f1`
        apptitle=`echo $i | cut -d= -f2- | cut -c1-52`
        echo "grep -l $apptitle /tmp/audiobooks/*AB.yaml | head -1"
        the_file=$(grep -l $apptitle /tmp/audiobooks/*AB.yaml | head -1)
        echo "the_file=$the_file"
        if [ -z $the_file ]
        then
                echo "No hiera file found for $apptitle ... skipping"
                continue
        fi
        appname=`basename $the_file .yaml`
        echo "url is[$url] and apptitle is [$apptitle] appname is [$appname]"
        exit 0
done
IFS=$SAVEIFS

The output the script produces is this:

pair http://www.example.com/product/B06XK9FGYD='A Soldier Of The Legion'
grep -l 'A Soldier Of The Legion' /tmp/audiobooks/*AB.yaml | head -1
the_file=
No hiera file found for 'A Soldier Of The Legion' ... skipping
pair http://www.example.com/product/B01GWQI0OS='Art of War Sun Tzu'
grep -l 'Art of War Sun Tzu' /tmp/audiobooks/*AB.yaml | head -1
the_file=
No hiera file found for 'Art of War Sun Tzu' ... skipping
pair http://www.example.com/product/B0717333MM='Bartleby, the Scrivener (version 2)'
grep -l 'Bartleby, the Scrivener (version 2)' /tmp/audiobooks/*AB.yaml | head -1
the_file=/tmp/audiobooks/BartlebyTheScrivener_AMZAD_AB.yaml
url is[http://www.example.com/product/B0717333MM] and apptitle is ['Bartleby, the Scrivener (version 2)'] appname is [BartlebyTheScrivener_AMZAD_AB]

The odd things is that each of the grep commands I echo out work when I run them from the command line ... for example:

$ grep -l 'A Soldier Of The Legion' /tmp/audiobooks/*AB.yaml | head -1
/tmp/audiobooks/A_Soldier_of_the_Legion_AB.yaml

And the script works for the title, 'Bartleby, the Scrivener (version 2)'.

Upvotes: 3

Views: 1943

Answers (3)

Eliah Kagan
Eliah Kagan

Reputation: 1704

$apptitle contains opening and closing single quote (') characters around the title, and your script passes them to grep.

  • Reading your script, this is easy to miss, because the commands your script tells the user it is running--and that you are running manually--allow the shell to perform quote removal, and thus do not pass those characters to grep. This is to say that your script is not running the same grep commands you want it to run.
  • If some of your .yaml files have single quotes around the title being searched for, the script will work for those files. But it will not work for the others. (I expect that is the reason it happens to work for one of your titles.)

Judging by your question's early edit history, it appears you might have considered this possibility and attempted to prevent it by changing

the_file=$(grep -l "$apptitle" /tmp/audiobooks/*AB.yaml | head -1)

to instead read:

the_file=$(grep -l $apptitle /tmp/audiobooks/*AB.yaml | head -1)

That won't help. The 's are still there. (I suggest undoing that change, though it is likely okay because of the value you've assigned IFS.)

When a quoting character like ' appears in the result of parameter expansion ($apptitle), the shell does not treat it specially. It does not prevent word splitting, nor is it subject to quote removal.

For example, when IFS has its default value, the output of x="'a b c'"; printf '(%s)' $x is ('a)(b)(c'), not (a b c). This is to say that, when x holds the value 'a b c', an unquoted $x expands to 'a b c', and word splitting turns that into 'a b c'.

In your case, you have changed IFS so splitting only happens on newlines and backspaces. grep matches lines, so you won't get a newline in a title. Assuming titles don't ever contain backspaces, it's fine (though perhaps stylistically confusing) to keep $apptitle unquoted. But this doesn't remove the enclosing ' characters.

Upvotes: 1

janos
janos

Reputation: 124656

If this line:

echo "grep -l $apptitle /tmp/audiobooks/*AB.yaml | head -1"

Produces output like this:

grep -l 'A Soldier Of The Legion' /tmp/audiobooks/*AB.yaml | head -1

Then that implies that the value of apptitle includes single-quotes.

You can try this to understand what's happening:

value1='A Soldier Of The Legion'
value2="'A Soldier Of The Legion'"
echo "$value1"
echo "$value2"

Outputs:

A Soldier Of The Legion
'A Soldier Of The Legion'

In other words, what the script really executes is this:

grep -l "'A Soldier Of The Legion'" /tmp/audiobooks/*AB.yaml | head -1

Which will only match if the yaml files contain the titles surrounded by single-quotes.

You probably want to strip the single-quotes from apptitle, for example:

apptitle=$(echo $i | cut -d= -f2- | cut -c1-52 | sed -e "s/^'//" -e "s/'$//")

The sed above will strip the single-quote at each end, and leave other single quotes in the middle of the string alone.

Upvotes: 3

Akif
Akif

Reputation: 6786

I think you need to place . (dot) when executing get-pairs.sh. Preceding dot means to "source" the contents of that file into the current shell.

for i in `. $get_pairs`

Upvotes: 0

Related Questions