CODEWITHSUNDEEP
pythonnltk
user14090797
user14090797

Reputation:

Can someone help me with error in using nltk word_tokenize function?

I have nltk installed on my mac. Below is my simple code to use nltk word_tokenize function. And I'm getting this error. Please advise!

import nltk

sentence = "I kN2ow Y1Ou."

s = sentence.lower()
words = nltk.word_tokenize(s)

print(words)

Error:

**********************************************************************
  Resource punkt not found.
  Please use the NLTK Downloader to obtain the resource:

  >>> import nltk
  >>> nltk.download('punkt')
  
  For more information see: https://www.nltk.org/data.html

  Attempted to load tokenizers/punkt/PY3/english.pickle

  Searched in:
    - '/Users/moeheinag/nltk_data'
    - '/Library/Frameworks/Python.framework/Versions/3.8/nltk_data'
    - '/Library/Frameworks/Python.framework/Versions/3.8/share/nltk_data'
    - '/Library/Frameworks/Python.framework/Versions/3.8/lib/nltk_data'
    - '/usr/share/nltk_data'
    - '/usr/local/share/nltk_data'
    - '/usr/lib/nltk_data'
    - '/usr/local/lib/nltk_data'
    - ''
**********************************************************************

So I just ran in terminal:

import nltk
nltk.download('punkt')

And received this errors:

[nltk_data] Error loading punkt: <urlopen error [SSL:
[nltk_data]     CERTIFICATE_VERIFY_FAILED] certificate verify failed:
[nltk_data]     unable to get local issuer certificate (_ssl.c:1124)>
False

Upvotes: 0

Views: 324

Answers (1)

Vaebhav
Vaebhav

Reputation: 5052

You need to download the punkt module as stated

Open Terminal on your mac , execute - python , then the below commands

nltk uses pre-trained , word and sentence tokenizers , which needs to be downloaded seperately

>>> import nltk
>>> nltk.download('punkt')

If in case the download fails , use below , reference

import nltk
import ssl

try:
    _create_unverified_https_context = ssl._create_unverified_context
except AttributeError:
    pass
else:
    ssl._create_default_https_context = _create_unverified_https_context

nltk.download()

Upvotes: 1

Related Questions