CODEWITHSUNDEEP
phpunicode
MTK
MTK

Reputation: 3580

php split string in chunks ngrams unicode char issue

I try to generate n grams from string in PHP For that I use this function from : https://gist.github.com/Xeoncross/5366393

function Bigrams($word){
    $ngrams = array();
    $len = strlen($word);
    for($i=0;$i+1<$len;$i++){
        $ngrams[$i]=$word[$i].$word[$i+1];
    }
    return $ngrams;
}

$word = "abcdefg";

print_r(Bigrams($word));

That OK return as expected ngrams :

[0] => ab
[1] => bc
[2] => cd
[3] => de
[4] => ef
[5] => fg

But for certain Unicode characters not return as expected:

Ex: for $word = "Lòria" return:

[0] => L�
[1] => ò
[2] => �r
[3] => ri

Or for $word = "пожалуйста" return:

[0] => п
[1] => ��
[2] => о
[3] => ��
[4] => ж
[5] => ��
[6] => а
[7] => ��
[8] => л

Any idea how to solve this?

Upvotes: 0

Views: 370

Answers (1)

splash58
splash58

Reputation: 26153

use unicode oriented string functions 

function Bigrams($word){
    $ngrams = array();
    $len = mb_strlen($word);
    for($i=0;$i+1<$len;$i++){
        $ngrams[$i]=mb_substr($word, $i, 2);
    }
    return $ngrams;
}

$word = "пожалуйста";

print_r(Bigrams($word));

result

Array
(
    [0] => по
    [1] => ож
    [2] => жа
    [3] => ал
    [4] => лу
    [5] => уй
    [6] => йс
    [7] => ст
    [8] => та
)

Upvotes: 1

Related Questions