perl regular expression to replace non printable characters in the form of &#xx;

Question

The XML character set is limited to the following:

[\x09\x0A\x0D\x20-\x{D7FF}\x{E000}-\x{FFFD}\x{10000}-\x{10FFFF}]

Entities may not be used to represent characters outside this set either.

I am parsing some XML data files from an external source using XML::DOM. Some of the XML files have non-printable characters encoded in the form of &#xx; (eg. ) which is causing the parser to crash as these are invalid. I am trying to find an easy way to remove these invalid characters. I have tried

$xml =~ s/(&#\c\c;)//g;

which doesn't seem to work. SO doesn't seem to have anything related and I have been searching online for a while with no success.

Answer 1

Try this one.

$xml =~ s{&#[0-9a-z]{1-2};}{}igs;

Answer 2

It makes sense to write a subtitution that finds all entities in the HTML and uses an /e modifier so that the replacement string can be supplied by a block of Perl code.

This example creates the $html_chars regex pattern from your own question that will check whether any single character is within range, and then uses it to test the values of all character entities in the string.

Note that the hash # in the pattern must be escaped as a consequence of the /x modifier that allows whitespace and comments to make the regex more readable.

My test string uses entities for all ASCII character codes in both decimal and hex, and you can see that the substitution removes just the control characters except for HT, LF and CR.

use strict;
use warnings;

my $html_chars = qr/[\x09\x0A\x0D\x20-\x{D7FF}\x{E000}-\x{FFFD}\x{10000}-\x{10FFFF}]/;

my $html = do {
   local $/;
   <DATA>;
};

$html =~ s{ ( &\# ( x[0-9A-Z]+ | [0-9]+ ) ; ) } {
   my ($entity, $code) = ($1, $2);
   $code = hex $code if $code =~ s/x//i;
   chr($code) =~ $html_chars ? $entity : '';
}eixg;

print $html;

__DATA__

Decimal
&#0;&#1;&#2;&#3;&#4;&#5;&#6;&#7;&#8;&#9;&#10;&#11;&#12;&#13;&#14;&#15;
&#16;&#17;&#18;&#19;&#20;&#21;&#22;&#23;&#24;&#25;&#26;&#27;&#28;&#29;&#30;&#31;
&#32;&#33;&#34;&#35;&#36;&#37;&#38;&#39;&#40;&#41;&#42;&#43;&#44;&#45;&#46;&#47;
&#48;&#49;&#50;&#51;&#52;&#53;&#54;&#55;&#56;&#57;&#58;&#59;&#60;&#61;&#62;&#63;
&#64;&#65;&#66;&#67;&#68;&#69;&#70;&#71;&#72;&#73;&#74;&#75;&#76;&#77;&#78;&#79;
&#80;&#81;&#82;&#83;&#84;&#85;&#86;&#87;&#88;&#89;&#90;&#91;&#92;&#93;&#94;&#95;
&#96;&#97;&#98;&#99;&#100;&#101;&#102;&#103;&#104;&#105;&#106;&#107;&#108;&#109;&#110;&#111;
&#112;&#113;&#114;&#115;&#116;&#117;&#118;&#119;&#120;&#121;&#122;&#123;&#124;&#125;&#126;&#127;

Hex
&#x00;&#x01;&#x02;&#x03;&#x04;&#x05;&#x06;&#x07;&#x08;&#x09;&#x0A;&#x0B;&#x0C;&#x0D;&#x0E;&#x0F;
&#x10;&#x11;&#x12;&#x13;&#x14;&#x15;&#x16;&#x17;&#x18;&#x19;&#x1A;&#x1B;&#x1C;&#x1D;&#x1E;&#x1F;
&#x20;&#x21;&#x22;&#x23;&#x24;&#x25;&#x26;&#x27;&#x28;&#x29;&#x2A;&#x2B;&#x2C;&#x2D;&#x2E;&#x2F;
&#x30;&#x31;&#x32;&#x33;&#x34;&#x35;&#x36;&#x37;&#x38;&#x39;&#x3A;&#x3B;&#x3C;&#x3D;&#x3E;&#x3F;
&#x40;&#x41;&#x42;&#x43;&#x44;&#x45;&#x46;&#x47;&#x48;&#x49;&#x4A;&#x4B;&#x4C;&#x4D;&#x4E;&#x4F;
&#x50;&#x51;&#x52;&#x53;&#x54;&#x55;&#x56;&#x57;&#x58;&#x59;&#x5A;&#x5B;&#x5C;&#x5D;&#x5E;&#x5F;
&#x60;&#x61;&#x62;&#x63;&#x64;&#x65;&#x66;&#x67;&#x68;&#x69;&#x6A;&#x6B;&#x6C;&#x6D;&#x6E;&#x6F;
&#x70;&#x71;&#x72;&#x73;&#x74;&#x75;&#x76;&#x77;&#x78;&#x79;&#x7A;&#x7B;&#x7C;&#x7D;&#x7E;&#x7F;

output

Decimal
	


 !"#$%&'()*+,-./
0123456789:&#59;<=>?
@ABCDEFGHIJKLMNO
PQRSTUVWXYZ[\]^_
`abcdefghijklmno
pqrstuvwxyz{|}~

Hex
	


 !"#$%&'()*+,-./
0123456789:&#x3B;<=>?
@ABCDEFGHIJKLMNO
PQRSTUVWXYZ[\]^_
`abcdefghijklmno
pqrstuvwxyz{|}~

Answer 3

I would recommending explicitly specifying which characters that you want to remove.

The following removes the unprintable character entities in the ascii range. This could easily be expanded if you wanted to cover all of the unprintable entities as you've defined them.

Also, please note as @ikegami mentioned in the question comments that using a regex like this will break the contents of CDATA section.

use strict;
use warnings;

my $data = do {local $/; <DATA>};

# Allowed entities:
# [\x09\x0A\x0D\x20-\x{D7FF}\x{E000}-\x{FFFD}\x{10000}-\x{10FFFF}]

# Decimal Character Entities
$data =~ s/&#0*(?!(?:9|1[03])\b)(?:[12]?[0-9]|3[01]);//g;

# Hex Character Entities
$data =~ s/&#x0*(?![9ADad]\b)1?[[:xdigit:]];//g;

print $data;

__DATA__
<?xml version="1.0" encoding="UTF-8" ?> 
<root>
    <hex_character_entities>
        <hex00>&#x00;&#x01;&#x02;&#x03;&#x04;&#x05;&#x06;&#x07;&#x08;&#x09;&#x0A;&#x0B;&#x0C;&#x0D;&#x0E;&#x0F;</hex00>
        <hex10>&#x10;&#x11;&#x12;&#x13;&#x14;&#x15;&#x16;&#x17;&#x18;&#x19;&#x1A;&#x1B;&#x1C;&#x1D;&#x1E;&#x1F;</hex10>
        <hex20>&#x20;&#x21;...</hex20>
    </hex_character_entities>
    <decimal_character_entities>
        <dec00>&#00;&#01;&#02;&#03;&#04;&#05;&#06;&#07;&#08;&#09;</dec00>
        <dec10>&#10;&#11;&#12;&#13;&#14;&#15;&#16;&#17;&#18;&#19;</dec10>
        <dec20>&#20;&#21;&#22;&#23;&#24;&#25;&#26;&#27;&#28;&#29;</dec20>
        <dec30>&#30;&#31;&#32;&#33;...</dec30>
    </decimal_character_entities>
</root>

Outputs:

<?xml version="1.0" encoding="UTF-8" ?> 
<root>
    <hex_character_entities>
        <hex00>&#x09;&#x0A;&#x0D;</hex00>
        <hex10></hex10>
        <hex20>&#x20;&#x21;...</hex20>
    </hex_character_entities>
    <decimal_character_entities>
        <dec0>&#09;</dec0>
        <dec1>&#10;&#13;</dec1>
        <dec2></dec2>
        <dec3>&#32;&#33;...</dec3>
    </decimal_character_entities>
</root>

Answer 4

I would try using \w instead of \c.

The following produces the correct results for me:

my $xml = <<XML;
<?xml version="1.0" encoding="UTF-8" ?> 
<outer>
  <inner>&#15;</inner>
</outer>
XML

$xml =~ s/&#\w{2};//g;