Does MySql full text search works reasonably with non-Latin languages (Hebrew, Arabic, Japanese...)

Question

Does MySql full text search works reasonably with non-Latin languages? (Hebrew, Arabic, Japanese...)

Addition: Did some tests... It has some problems with Hebrew. Example: The name מוסינזון is pronounced the same as מושינזון but searching one won't find the other, as this is a common spelling error in Hebrew, it seems I will have to do some data manipulation for it to work perfectly.

Answer 1

Yes MySQL fulltext search works well for Arabic. Just make sure of the following where needed:

1. COLLATION = utf8_unicode_ci & CHARACTER SET = utf8. (Databases, Tables, and Columns).
2. Index words of 3 letters and more. This is Very Important for Arabic, ft_min_word_len = 3 (see show variables like "ft_%";)
3. Check the version of MySQL (5.5 or 5.6), and Engine (InnoDb or MyIsam)

Answer 2

Though Hebrew support in MySQL is limited, your problem is more a problem of people using incorrect spelling, then a dysfunction of the MySQL server in this perspective. When you misspell a word in Google, it will show you a suggestion and you can click on that suggestion to search for that term.

Perhaps you could build some program that has the same behaviour, e.g. you could create a table that has 2 fields, one containing the commonly misspelled word and the other containing the correct spelling. You could then build a program that finds the misspelled word and displays the suggestion.

Answer 3

Japanese and Chinese use their own whitespace symbols that MySQL does not understand.

Make sure that the words in the texts you are going to index are separated with ASCII separators (spaces, commas etc). Anything outside the ASCII range will probably not work.

Besides, you'll probably need to fix ft_min_word_len: by default, MySQL won't index words shorter than 4 characters, and most Japanese and Chinese words.

In Cyrillic languages transilteration errors are quite common.

All letters from this sequence: АВЕКМНОРСТуХ / ABEKMHOPCTyX are indistinguishable in most fonts.

The worst of them is Cyrillic С/ Latin C: both these symbols are located on one key on the keyboard and do not differ at all in most fonts, but they have different codes.

MySQL will not catch it either.

Answer 4

Yes, however, check out what stopwords are.

Answer 5

So long as your collation is set properly, it works splendidly.

Unicode will work for most of this, of course. But that doesn't really translate Latin characters to them very well (for example, in a Dutch collation aa will be recognized as å).