CODEWITHSUNDEEP
bashperl
MAS
MAS

Reputation: 96

Reading and writing line by line in a bash script

After searching online I was able to figure out how to read a file line by line:

while read p; do
  echo $p
done < file.txt

But I would actually like to modify the line in the file. For example:

while read p; do
  if condition
  then
    echo $p | perl -i -pe 's/a/b/'
  fi
done < file.txt

However this doesn't actually modify the file.

Upvotes: 3

Views: 1494

Answers (1)

zdim
zdim

Reputation: 66964

Update   A far better version of bash code added. Thanks to Charles Duffy for comments.

Your Perl one-liner takes a line piped into it by echo $p |, getting its standard input that way. It doesn't do anything with the file itself, so the -i flag has no effect. The -p makes it print to the standard output stream. So that whole line, echo ..., doesn't touch the file.

You can redirect the output to a new file and then move that to overwrite file.txt. Here is a simple minded example, that appends each line to a new file. For better bash code see the update below.

while read p; do
    if condition
    then
        echo $p | perl -pe 's/a/b/' >> temp_out.txt
    else
        echo $p >> temp_out.txt
    fi
done < file.txt
mv temp_out.txt file.txt

We have to add the else where all unmodified lines are also appended. Note that in general we cannot have just some lines replaced but the whole file has to be re-written.

If this is all that the script does you can do it with a very simple one-liner, see the end. If more work is done you can also put it all in a Perl script but I take it that there may be other good reasons for a bash script.

Update   A much better version of the above. See read and echo in Builtins in Bash manual

  • Appending each line opens the file anew each time without a need for that. Just redirect at the end of the loop, much like it is done in the terminal

  • read uses backslash for escaping, removing it from input. Turn that off with -r

  • Trailing white space is removed, as a part of breaking the line into words. Suppress this by unsetting the variable that controls which characters are used for splitting, IFS=

  • The echo $p can do all kinds of unintended things. A formatted print is better, printf '%s\n' "$p", or at least echo "$p"

With this,

while IFS= read -r p; do
    if condition
    then
        echo "$p" | perl -pe 's/a/b/'
    else
        echo "$p"
    fi
done < file.txt  > temp_out.txt
mv temp_out.txt file.txt

Finally, if the sole purpose of the Perl one-liner were to run a simple substitution, it is much better to simply do that in the shell itself than to have a pipeline and run a whole new process for each line.

echo "${p//a/b}"

Thanks to Charles Duffy for raising all these points in comments.

A few comments on Perl one-liners. See documentation at perlrun.

The command perl -e '...' executes any valid Perl code between ''. When we add the -n or -p switch it also reads standard input and executes that code on a line of it at the time, where -p also prints out each line after it's processed. The standard input can be supplied to it from a file,

perl -pe '...' input.txt

in which case adding -i flag will result in the file being changed in-place. Or, the input can be piped into it, for example

echo "input text" | perl -pe '...'

in which case the processed line is printed to standard output. This can be redirected to a file, as in the answer above.

To make changes to a given file a line at a time you only need this on the command line

perl -i -pe 's/a/b/' file.txt

If there is more work to do then it may well be better to put it in a script, of course. In this case the one-liner can be a command in the bash script as well, replacing all that code above (unless some bash-specific functionality is preferred for processing lines).

Upvotes: 3

Related Questions