CODEWITHSUNDEEP
phpregex
Niteriter
Niteriter

Reputation: 140

Regex: how to get \b to work with utf-8 charset?

I'm trying to remove words shorter than 3 characters from string, and came up with this regex: !\b[\pL\pN]{1,2}\b!u which works in general, but breaks on most of polish characters (ąćęłńśżź), which are treated as non letter symbols and word boundaries (and splits words in the middle if offending char happen to be there).

Sample code to demonstrate this phenomenon:

$text = " ää äaa aaäaaa ââ âaa aaâaa ąą ąaa aaąaaa àà áá ââ ãã åå ää ææ ßß ćć èè êê ëë ęę éé ìì íí îî ïï öö òò ôô õõ øø óó ùù úú ûû üü ąą ćć ęę łł ńń óó śś żż źź ";
echo "<h1>input</h1><p>$text</p>";
$text = preg_replace('!\b[\pL\pN]{1,2}\b!u', 'X', $text);
echo "<h1>output</h1><p>$text</p>";

Gives output (note aaąaaa being split):

input
ää äaa aaäaaa ââ âaa aaâaa ąą ąaa aaąaaa àà áá ââ ãã åå ää ææ ßß ćć èè êê ëë ęę éé ìì íí îî ïï öö òò ôô õõ øø óó ùù úú ûû üü ąą ćć ęę łł ńń óó śś żż źź 

output
X äaa aaäaaa X âaa aaâaa ąą ąX XXaaa X X X X X X X X ćć X X X ęę X X X X X X X X X X X X X X X ąą ćć ęę łł ńń X śś żż źź
                                 ^split

Setting different locale with setlocale() didn't help.

Using PHP 5.3.0 on Windows.

Upvotes: 2

Views: 1349

Answers (3)

Dmitrij Golubev
Dmitrij Golubev

Reputation: 694

I am using following regex.

$regExp = '(?<=[\s\p{P}\p{S}\p{C}]|^)('.$search.')(?=[\s\p{P}\p{S}\p{C}]|$)';

On my experience simple excluding ([^\p{L}\p{N}]) will not work, because some languages (like hindi/devanagari or japanese) have hidden symbols (vovels symbols), that shows as part of previous or next symbol.

Upvotes: 2

mario
mario

Reputation: 145482

You cannot make \b work. It only matches ASCII word boundaries. http://www.mail-archive.com/[email protected]/msg00110.html

But you could try to use negative assertions (?<!\pL) and (?!\pL) in place of the \b test.

Upvotes: 6

NikiC
NikiC

Reputation: 101936

Try:

'~(?<=[\PL\PN]|^)[\pL\pN]{1,2}(?=[\PL\PN]|$)~u'

This matches any letter or number, which is

  • preceded by something that isn't a letter or number or by the start of the string
  • followed by something that isn't a letter or number or by the end of the string

Upvotes: 3

Related Questions