CODEWITHSUNDEEP
rubyutf-8character-encodingweb-crawler
Harue
Harue

Reputation: 149

Error with utf8 encoding

When I get data from some website, sometime the data is encode in utf8 but look like this:

Thỏ , Nạt

The accent mark is seperated from character when in fact these string must be:

Thỏ, Nạt

I don't know what is the problem here and how to correct it. Can someone help me with this

Upvotes: 1

Views: 240

Answers (1)

Jukka K. Korpela
Jukka K. Korpela

Reputation: 201866

The first sample string contains two Vietnamese characters in decomposed form. The first one of them is “ỏ”, consisting of simple letter “o” followed by U+0309 COMBINING HOOK ABOVE.

The second sample string has those characters in precomposed form. The first one of them is “ỏ” U+1ECF LATIN SMALL LETTER O WITH HOOK ABOVE.

The decomposed and precomposed form are defined to be “canonical equivalent” and are normally expected to result in the same rendering (though this does not always happen). They are not identical, however; in programmatic comparison of characters and strings, they are very much different.

Mostly Latin letters with diacritics, such as “é” and “ä”, are used in precomposed form only, since that’s what keyboard drivers, online keyboards, character picking utilities, etc., normally produce. However, Vietnamese keyboard drivers often work so that some diacritic marks are entered after entering a base character, and the diacritic is thus produced as a combining character, i.e. the letter (like “ỏ”) is then in decomposed form.

One way of dealing with this issue, recommended in many contexts, is to convert your strings to Normalization Form C (NFC). This would put these characters into precomposed form. Note, however, that conversion to NFC removes some other distinctions, too (but this is not relevant if the text is in Vietnamese only and does not contain special symbols).

It remains a mystery why the first sample string has a space character before the comma.

Upvotes: 4

Related Questions