CODEWITHSUNDEEP
pythonerror-handlingcompiler-errorsnltk
user3422952
user3422952

Reputation: 317

Python (nltk) - UnicodeDecodeError: 'ascii' codec can't decode byte

I'm new to NLTK. I'm getting this error and I've searched around for encoding/decoding and specifically the UnicodeDecodeError but this error seems specific to the NLTK source code.

Here's the error:

Traceback (most recent call last):
  File "A:\Python\Projects\Test\main.py", line 2, in <module>
    print(pos_tag(word_tokenize("John's big idea isn't all that bad.")))
  File "A:\Python\Python\lib\site-packages\nltk\tag\__init__.py", line 100, in pos_tag
    tagger = load(_POS_TAGGER)
  File "A:\Python\Python\lib\site-packages\nltk\data.py", line 779, in load
    resource_val = pickle.load(opened_resource)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xcb in position 0: ordinal not in range(128)

How do I go around fixing this error?

Here's what causes the error:

from nltk import pos_tag, word_tokenize
print(pos_tag(word_tokenize("John's big idea isn't all that bad.")))

Upvotes: 9

Views: 6965

Answers (4)

LuckyMatina
LuckyMatina

Reputation: 41

I had the same problem with you. I use Python 3.4 in Windows 7.

I had installed the "nltk-3.0.0.win32.exe" (from here). But when i installed the "nltk-3.0a4.win32.exe" (from here), my problem with nltk.pos_tag was solved. Check it.

EDIT: If the second link doesn't work, you can look here.

Upvotes: 4

Simone Dagli Orti
Simone Dagli Orti

Reputation: 476

try this... NLTK 3.0.1 with Python 2.7.x

import io
f = io.open(txtFile, 'rU', encoding='utf-8')

Upvotes: 5

Shivamshaz
Shivamshaz

Reputation: 280

Try using the module "textclean" 

>>> pip install textclean

Python code

from textclean.textclean import textclean
text = textclean.clean("John's big idea isn't all that bad.")
print pos_tag(word_tokenize(text))

Upvotes: -2

Dave
Dave

Reputation: 1

Duplicate: NLTK 3 POS_TAG throws UnicodeDecodeError

Long story short: NLTK isn't compatible with Python 3. You have to use NLTK 3 which sounds a bit experimental at this point.

Upvotes: -2

Related Questions