CODEWITHSUNDEEP
pythonmongodbfull-text-searchpymongo
William
William

Reputation: 4408

Pymongo full text search in chinese

I know Pymongo supports text search feature for many languages e.g. English, French, German..., but it doesn't work for Chinese. Is there a available way to implement Chinese full text search, under the environment of MongoDB 3.4 + Pymongo 3.4 ?

Upvotes: 1

Views: 1090

Answers (1)

William
William

Reputation: 4408

I think the main reason that pymongo free version doesn't support Chinese full text search is that Chinese segmentation is difficult. I have an indirect approach to tackle Chinese full text search by Pymongo. One can finish the segmentation before he store the corpus into MongoDB or search a sentence from MongoDB. The jieba module is what I strongly recommends for Chinese segmentation. There is a self-contained simple example, which works for me to some extent. 

from pymongo import MongoClient
from pymongo import TEXT
import jieba
client = MongoClient()
dialogs = client['db']['dialogs_zh_fulltext']
d1 = {
    'text_in': '你 早上 吃 的 什么 ?',
    'text_out': '我 吃 的 鸡蛋',
}
d2 = {
    'text_in': '你 今天 准备 去 哪 ?',
    'text_out': '我 要 回家',
}
dialogs.insert_many([d1,d2])
dialogs.create_index([('text_in', TEXT)], default_language='en')
keywords = ' '.join(jieba.lcut('你今天早上去哪了?'))
print('keywords: {}'.format(keywords))
cursor = dialogs.find({'$text': {'$search':keywords}}, {'score':{'$meta':'textScore'}})
for x in cursor.sort([('score', {'$meta':'textScore'})]):
    print(x)

OUTPUT:

keywords: 你 今天 早上 去 哪 了 ?
{'_id': ObjectId('59673a0d5975ae05e9b27dd8'), 'text_in': '你 今天 准备 去 哪 ?', 'text_out': '我 要 回家', 'score': 2.4}
{'_id': ObjectId('59673a0d5975ae05e9b27dd7'), 'text_in': '你 早上 吃 的 什么 ?', 'text_out': '我 吃 的 鸡蛋', 'score': 1.2}

Upvotes: 3

Related Questions