How to Circumvent Perl's string escaping the replacement string in s/// when it's read from a file?

Question

This question is similar to my last one, with one difference to make the toy script more similar to my actual one.

Here is the toy script, replace.pl (Edit: now with 'use strict;', etc)

#! /usr/bin/perl -w

use strict;

open(REPL, "<", $ARGV[0]) or die "Couldn't open $ARGV[0]: $!!";
my %replacements;
while(<REPL>) {
   chomp;
   my ($orig, $new, @rest) = split /,/;
   # Processing+sanitizing of orig/new here
   $replacements{$orig} = $new;
}
close(REPL) or die "Couldn't close '$ARGV[0]': $!";

print "Performing the following replacements\n";
while(my ($k,$v) = each %replacements) {
   print "\t$k => $v\n";
}

open(IN, "<", $ARGV[1]) or die "Couldn't open $ARGV[1]: $!!";
while ( <IN> ) {
   while(my ($k,$v) = each %replacements) {
      s/$k/$v/gee;
   }
   print;
}
close(IN) or die "Couldn't close '$ARGV[1]': $!";

So, now lets say I have two files, replacements.txt (using the best answer from the last question, plus a replacement pair that doesn't use substitution):

(f)oo,q($1."ar")
cat,hacker

and test.txt:

foo
cat

When I run perl replace.pl replacements.txt test.txt I would like the output to be

far
hacker

but instead it's '$1."ar"' (too much escaping) but the results are anything but (even with the other suggestions from that answer for the replacement string). The foo turns into ar, and the cat/hacker is eval'd to the empty string, it seems.

So, what changes do I need to make to replace.pl and/or replacements.txt? Other people will be creating the replacements.txt's, so I'd like to make that file as simple as possible (although I acknowledge that I'm opening the regex can of worms on them).

If this isn't possible to do in one step, I'll use macros to enumerate all possible replacement pairs for this particular file, and hope the issue doesn't come up again.

Answer 1

Get rid of the q() in the replacement string;

Should be just
(f)oo,$1."ar"
as in ($k,$v) = split /,/, $_;

Warning: using external input data in evals is very, very dangerous

Or, just make it
(f)oo,"${1}ar"

No modification to the code is necessary either way e.g. s///gee.

Edit @drhorrible, if it doesen't work then you have other problems.

use strict;use warnings;

my $str = "foo";
my $repl = '(f)oo,q(${1}."ar")';
my ($k,$v) = split /,/, $repl;
$str =~ s/$k/$v/gee;
print $str,"\n";

$str = "foo";
$repl = '(f)oo,$1."ar"';
($k,$v) = split /,/, $repl;
$str =~ s/$k/$v/gee;
print $str,"\n";

$str = "foo";
$repl = '(f)oo,"${1}ar"';
($k,$v) = split /,/, $repl;
$str =~ s/$k/$v/gee;
print $str,"\n";

output:
${1}."ar"
far
far

Answer 2

Please don't give us non-working toy scripts that don't use strict and warnings. Because one of the first things people will do in debugging is to turn those on, and you've just caused work.

Second tip, use the 3-argument version of open rather than the 2-argument version. It is safer. Also in your error checking do as perlstyle says (see http://perldoc.perl.org/perlstyle.html for the full advice) and include the file name and $!.

Anyways your problem is that the code you were including was q($1."ar"). When executed this returns the string $1."ar". Get rid of the q() and it works fine. BUT it causes warnings. That can be fixed by moving the quoting into the replace script, and out of the original script.

Here is a fixed script for you:

#! /usr/bin/perl -w
use strict;

open(REPL, "<", $ARGV[0]) or die "Couldn't open '$ARGV[0]': $!!";
my %replacements;
while(<REPL>) {
   chomp;
   my ($orig, $new) = split /,/;
   # Processing+sanitizing of orig/new here
   $replacements{$orig} = '"' . $new . '"';
}
close(REPL) or die "Couldn't close '$ARGV[0]': $!";

print "Performing the following replacements\n";
while(my ($k,$v) = each %replacements) {
   print "\t$k => $v\n";
}

open(IN, "<", $ARGV[1]) or die "Couldn't open '$ARGV[1]': $!!";
while ( <IN> ) {
   while(my($k,$v) = each %replacements) {
      s/$k/$v/gee;
   }
   print;
}
close(IN) or die "Couldn't close '$ARGV[1]': $!";

And the modified replacements.txt is:

(f)oo,${1}ar
cat,hacker

Answer 3

You have introduced one more level of interpolation since the last question. You can get the right result by either:

1. Lay a 3rd "e" modifier on your substitution

   s/$k/$v/geee;    # eeek

2. Remove a layer of interpolation in replacements.txt by making the first line

   (f)oo,$1."ar"