Multiline search replace with Perl

Question

I know this kind of questions have been asked already many times before. The reason why I come here again is that I feel like I've missed something simple and fundamental.

Is it possible to make this kind of search-replace routine better. For example without opening same file twice. Also speed related advices are welcome.

Please notice that this works with multiline matches and replaces also multiline strings.

#!/bin/perl -w -0777

local $/ = undef;

open INFILE, $full_file_path or die "Could not open file. $!";
$string =  <INFILE>;
close INFILE;

$string =~ s/START.*STOP/$replace_string/sm;

open OUTFILE, ">", $full_file_path or die "Could not open file. $!";
print OUTFILE ($string);
close OUTFILE;

Answer 1

I know this has been answered but this is how I managed to solve this.

Let's say you wanted to change out a UUID but there must be a match on the line above because you have many UUID's that belong to other things.

perl call in a bash script in Ubuntu 20:

_UUID=$(uuidgen | sed 's/-//g')
export _UUID
perl -0777 -pi.back -e 's/(<stringProp\sname="Argument\.name">_BINARYVIDEOTEMPURL<\/stringProp>\n.*<stringProp\sname="Argument\.value">)[a-zA-Z0-9]{32}(<\/stringProp>)/$1$ENV{_UUID}$2/g;' test.txt

Your test.txt file reads like so: (not a valid XML I know but just create it)

<?xml version="1.0" encoding="UTF-8"?> <jmeterTestPlan version="1.2" properties="5.0" jmeter="5.2.1">
  <hashTree>
<TestPlan guiclass="TestPlanGui" testclass="TestPlan" testname="K8S Load Test Plan" enabled="true">      
  <stringProp name="TestPlan.user_define_classpath"></stringProp>
</TestPlan>
      <collectionProp name="Arguments.arguments">
        <elementProp name="_SESSIONID" elementType="Argument">
          <stringProp name="Argument.name">_SESSIONID</stringProp>
          <stringProp name="Argument.value">7c096b65-84b6-40c9-be93-a5891ec0394d</stringProp>
          <stringProp name="Argument.metadata">=</stringProp>
        </elementProp>
        <elementProp name="_BINARYVIDEOTEMPURL" elementType="Argument">
          <stringProp name="Argument.name">_BINARYVIDEOTEMPURL</stringProp>
          <stringProp name="Argument.value">64e1886127fa41c4a58e59fe2bb098e1</stringProp>
          <stringProp name="Argument.metadata">=</stringProp>
        </elementProp>
      </collectionProp>

So a lot is happening here but let me explain.

1. Create a new UUID to replace.
2. Export the UUID because perl will pick it up in the ENVIRONMENT variables.
3. Call perl to handle the search and replacement

• -077 makes perl be able to use multiline and accomplish the multiline lookahead and behind. I couldn't tell you how perl works.
• -pi.back basically inline editing and backing up the file.
• -e is basically 's/reaplcethis/withthis/g' but it contains the regex with new lines needed to match. Plus it shows how to use ENVIRONMENT variables and grouping to recreate the string.

Anyways, hope this helps someone.

Answer 2

the combination of bash script + perl -pi -e is unbeatable - an example of bash function to directly type the search and replace strings before the EOF label :

# usage put into foobar.sh file, source foobar.sh file
# call directly into the shell do_multiline_srch_and_replace
do_multiline_srch_and_replace(){

                test -z $dir_to_work && {
         echo "You must export dir_to_work=<<the-dir>> - it is empty !!!"; exit 1;
      }
                test -d $dir_to_work || {
         echo "The dir to work on: \"$dir_to_work\" is not a dir !!!"; exit 1;
      }

                echo "INFO dir_to_work: $dir_to_work" ; sleep 1
                echo "INFO START :: searching and replacing in the non-binary files only"

                while read -r file ; do (
                        echo "DEBUG working on the following file: $file"

         # those pattern in the file names we want to skip usually - git, not , py files
         case "$file" in
            *.git*)
            continue ;;
            *node_modules*)
            continue ;;
            *.venv*)
            continue ;;
         esac
         # note the string should be exactly between the s|| and the replace str between the ||gs
         # the 'EOF' guarantees that no special chars from the shell will affect the result
                        perl -pi - <<'EOF' "$file"
BEGIN{undef $/;}
s|a multiline
string|the multiline
string to replace|gs
EOF
                );
                done < <(find $dir_to_work -type f -not -exec file {} \; | grep text | cut -d: -f1)

                echo "INFO STOP  :: search and replace in non-binary files"

}

Answer 3

you might want to check out my Perl script, which is battle tested (used heavily in production), and has quite a lot of features, such as:

• do multiple search-replace or query-search-replace operations
• search-replace expressions can be given on the command line or read from a file processes multiple input files
• recursively descend into directory and do multiple search/replace operations on all files
• user defined perl expressions are applied to each line of each input file optionally run in paragraph mode (for multi-line search/replace)
• interactive mode
• batch mode
• optionally backup files and backup numbering
• preserve modes/owner when run as root
• ignore symbolic links, empty files, write protected files, sockets, named pipes, and directory names
• optionally replace lines only matching / not matching a given regular expression

https://github.com/tilo/replace_string

Answer 4

Pulling the short answer from the comments, for anyone looking for a quick one-liner, and the reason Perl is ignoring their RegEx options from the command line.

perl -0pe 's/search/replace/gms' file

Without the -0 argument, Perl processes data line-by-line, which causes multiline searches to fail.

Answer 5

This kind of search and replace can be accomplished with a one-liner such as -

perl -i -pe 's/START.*STOP/replace_string/g' file_to_change

For more ways to accomplish the same thing check out this thread. To handle multi-line searches use the following command -

perl -i -pe 'BEGIN{undef $/;} s/START.*STOP/replace_string/smg' file_to_change

In order to convert the following code from a one-liner to a perl program have a look at the perlrun documentation.

If you really find the need to convert this into a working program then just let Perl handle the file opening/closing for you.

#!/usr/bin/perl -pi
#multi-line in place substitute - subs.pl
use strict;
use warnings;

BEGIN {undef $/;}

s/START.*STOP/replace_string/smg;

You can then call the script with the filename as the first argument

$perl subs.pl file_to_change

If you want a more meatier script where you get to handle the file open/close operations(don't we love all those 'die' statements) then have a look at the example in perlrun under the -i[extension] switch.

Answer 6

Considering that you slurp in the whole contents of the file with:

local $/ = undef;

open INFILE, $full_file_path or die "Could not open file. $!";
$string =  <INFILE>;
close INFILE;

And then do all the processing with $string, there's no connection between how you handle the file and how you process the contents. You'd have an issue if you opened the file for writing before you were done reading it, since opening a file for writing creates a new file, discarding the previous contents.

If all you're trying to do is save on open/close statements, then do as Jonathan Leffer suggested. If your question is about multiline search and replace, then please clarify what the problem is.