CODEWITHSUNDEEP
phphtmlregexunicodeutf-8
nEAnnam
nEAnnam

Reputation: 1256

"Word" characters from the Unicode Set

I was wondering how PCRE detects word characters from whatever language. I was testing this string:

"間違つ"

The php file is encoded as UTF-8 and is properly tagged with the Charset=UTF-8 in the Content type tag.

<?php

$string="\xE9\x96\x93\xE9\x81\x95\xE3\x81\xA4"; //Bytestream from "間違つ" 
$string=preg_replace('/\w/','\w',$string);
echo $string;
echo "<br>";


$byte="\xE9"; //I've tried with each byte separately to find word characters
if(preg_match('/\w/',$byte)){
    echo "$byte is a word";
    }
else{ 
    echo "$byte is not a word";
    }
?>

"\xE9" "\xE9" "\xE3" from all the bytes, are words.

It display:

Displayed

I know why the symbols appear. The Decoder use the Unicode replacement character, code point FFFD, as the decoding of an invalid UTF-8 sequence rather than stop processing the text. There are invalid sequences since one "word character" is replaced by the replacement '\w' and then it broke the "byte secuence" to show.

So the questions are:

why those characters are matched like words if they aren't valid UTF-8 sequences?

How to know wich characters are really word characters of all the Unicode Set?

Upvotes: 1

Views: 3198

Answers (2)

Gareth Rees
Gareth Rees

Reputation: 65854

I believe that your regular expression engine is interpreting your stream of bytes as if they were encoded in ISO Latin-1 (which they are not). In ISO Latin-1,

  • E3 is LATIN SMALL LETTER A WITH TILDE
  • E9 is LATIN SMALL LETTER E WITH ACUTE

which are "word" characters, but

which are not word characters.

You can set the /u modifier on the regular expression to request it to work with UTF-8 rather than Latin-1. See the PHP manual on pattern modifiers.

Upvotes: 2

Saxoier
Saxoier

Reputation: 1287

You have to set the u-Flag otherwise it is interpreted as a ISO-8859-1 string.

The following script shows which chars \w match without the u-Flag:

header("Content-Type: text/plain");
$i = 255;
while($i--)
{
    preg_match('/\w/S', chr($i), $m);
    printf("%' 1s \x%s\n", $m[ 0 ], strtoupper(bin2hex($m[ 0 ])));
}

Only [a-zA-Z] are matched by \w if u-Flag is set:

// added 'A' at the beginning and 'B' at the end
preg_match_all('/\w/u', "A\xE9\x96\x93\xE9\x81\x95\xE3\x81\xA4B", $m);
print_r($m);

Attention: If the u-Flag is present preg_* will silently fail to parse the string if it contains non-unicode-chars (e.g. \x80-\xFF).

Upvotes: 2

Related Questions