CODEWITHSUNDEEP
pythonutf-8arabicstemming
user2822966
user2822966

Reputation: 117

Python ISRIStemmer for Arabic text

I am running the following code on IDLE(Python) and I want to enter Arabic string and get the stemming for it but actually it doesn't work

>>> from nltk.stem.isri import ISRIStemmer
>>> st = ISRIStemmer()
>>> w= 'حركات'
>>> join = w.decode('Windows-1256')
>>> print st.stem(join).encode('Windows-1256').decode('utf-8')

The result of running it is the same text in w which is 'حركات' which is not the stem

But when do the following:

>>> print st.stem(u'اعلاميون')

The result succeeded and returns the stem which is 'علم'

Why passing some words to stem() function doesn't return the stem?

Upvotes: 6

Views: 10682

Answers (7)

mayssa-k
mayssa-k

Reputation: 1

from nltk import word_tokenize from nltk.stem.isri import ISRIStemmer

st = ISRIStemmer() word_list = "من طلب العلا سهر الليالي"

Define a function

def filter(word_list): wordsfilter=[] for a in word_tokenize(word_list): stem = st.stem(a) wordsfilter.append(stem) print(wordsfilter)

Call the function

filter(word_list)

Upvotes: -1

Taie
Taie

Reputation: 1189

Here is another example on how to use the stemmer (of course you can remove stop words first!)

from nltk import word_tokenize
from nltk.stem.isri import ISRIStemmer

st = ISRIStemmer()
word_list = "عرض يستخدم الى التفاعل مع المستخدمين في هاذا المجال !وعلمآ تكون الخدمه للستطلاع على الخدمات والعروض المقدمة"

# Define a function
def filter(word_list):
    wordsfilter=[]
    for a in word_tokenize(word_list):
        stem = st.stem(a)
        wordsfilter.append(stem)
    print(wordsfilter)

# Call the function
filter(word_list)

Here is the result:

['عرض', 'خدم', 'الى', 'فعل', 'مع', 'خدم', 'في', 'هذا', 'جال', '!', 'علمآ', 'تكون', 'خدم', 'طلع', 'على', 'خدم', 'عرض', 'قدم']

Upvotes: 1

user2822966
user2822966

Reputation: 117

Ok, I solved the problem by myself using the following:

w = 'حركات' 
st.stem(w.decode('utf-8'))

and it gives the root correctly which is "حرك"

Upvotes: 5

MD ZDN
MD ZDN

Reputation: 209

You can use this snippet to directly stem text:

from nltk import word_tokenize

from nltk.stem.isri import ISRIStemmer

st = ISRIStemmer()

w= " البحث العلمي أو البحث أو التجربة التنموية هو أسلوب منظم في جمع المعلومات الموثوقة وتدوين الملاحظات والتحليل الموضوعي لتلك المعلومات باتباع أساليب ومناهج علمية محددة بقصد التأكد من صحتها أو تعديلها أو إضافة الجديد لها، ومن ثم التوصل إلى بعض القوانين والنظريات والتنبؤ بحدوث مثل هذه الظواهر والتحكم في أسبابها"

for a in word_tokenize(w):

    print(st.stem(a))

Upvotes: 2

MZe
MZe

Reputation: 168

This code above won't work in Python 3 because we are trying to decode an object that is already decoded. So, there is no need to decode from UTF-8 anymore.

Here is the new code that should work just fine in Python 3. 

import nltk
from nltk.stem.isri import ISRIStemmer
st = ISRIStemmer()
w= 'حركات'
print(st.stem(w))

Upvotes: 8

Lakhdar Benzahia
Lakhdar Benzahia

Reputation: 83

there is a new light arabicstemmer here developed with snowball framework

Upvotes: 3

Chiraz BenAbdelkader
Chiraz BenAbdelkader

Reputation: 180

Well, just notice that your two strings actually only differ by a mere "u" at the beginning of the second string :

w = 'حركات'
w2 = u'اعلاميون'

But that tiny "u" made all the difference : w is a UTF8 string (default character encoding in Python), while w2 is a Unicode string.

Hence all you really need to do is make sure your string is defined as a Unicode string, and then you can use the stem function normally without any extra decoding step :

w = u'حركات'
print st.stem(w)

Upvotes: 0

Related Questions