CODEWITHSUNDEEP
bashshellunix
user1413813
user1413813

Reputation: 5

shell script to read file line by line does not work

i have this master script

>loanIds
>log
>count
mkdir -p duplicates
rm -rf duplicates/*
for file  in x*
do
./getXmlData.sh $file &
done

Source for getXmlData.sh

cat $file | while read line
do
echo $file" | "$line
done

i have 40 files that start with letter x each x* file has 1000 lines.

when i run the script, as you can see it will create 40 instances in the background and process each x* file

The problem i have is, ./getXmlData.sh does not echo all 1000 lines for each of those 40 files... for some files it echos all 1000 lines.. for some of them it echos only 10-15 lines

What could be the issue?

Upvotes: 0

Views: 1563

Answers (2)

dganesh2002
dganesh2002

Reputation: 2230

try this way ->

file=$1
chmod 755 $file
exec <$file
while IFS= read -r
do
    echo "$file | $REPLY"
done

Or to avoid issues with "read" command, I came up with below logic which basically calculates the number of lines and prints it one by one using head/tail command ->

file=$1 
lines=`wc -l $file | cut -d" " -f1`
for ((i=1;i<=$lines;i++));
do
    line=`head -$i $file | tail -1`
    echo "$file | $line"
done

Upvotes: 0

Jonathan Leffler
Jonathan Leffler

Reputation: 755114

Unless you've not shown an assignment in your getXmlData.sh script, the problem is likely that you are passing $file in the calling script but you need to reference it as $1 in the called script. That is, you should be writing:

cat "$1" |
while read -r line
do
    echo "$1 | $line"
done

or (better — avoiding UUOC or Useless Use of cat):

while read -r line
do
    echo "$1 | $line"
done < "$1"

or something similar. Indeed, we could even suggest:

sed "s%^%$1 | %" "$1"

This assumes that the file name doesn't contain a newline or a percent (but allows the name to include slashes, hence the use of % instead of / in the s/// command). Note that the file name is enclosed in double quotes each time it is used; this protects your script from file names containing spaces, etc. Also note that the echo encloses $line (as well as $1) in double quotes; this preserves the spacing in the data, and avoids other metacharacter expansions too. If you really wanted all sets of multiples spaces or tabs converted to a single space, then the original formulation was OK, but that is seldom the desired result.

One other possibility: maybe after the loop, you should wait for all the children to complete. You do that by telling the shell to:

wait

It won't progress further until its background processes terminate.

Upvotes: 3

Related Questions