CODEWITHSUNDEEP
pythonpython-3.xnltk
Jerry
Jerry

Reputation: 15

Word_tokenize twitter data

I am trying to just get the english words out of the text file for a simple word frequency objective. How can I filter out the other strings in the list? 

from nltk.tokenize import word_tokenize
words = word_tokenize(message.replace('\n',' '))

print(words)

giving output like this: 

['Amazon', 'b', 'maji_opai', 'am\\xcd\\x9ca\\xcd\\x89zon\\xe2\\x80\\xa6', '\\xcb\\x99\\xea\\x92\\xb3\\xe2\\x80\\x8b\\xcb\\x99', 'Amazon', "b'RT", 'WorkingGIrl', 'For', 'people', 'love', 'REAL', 'paperbacks', 'THE', 'PARIS', 'EFFECT', '10', 'right', 'https', '//', 'https', 'Amazon', "b'RT", 'AbsentiaSeries', 'ABSENTIA', 'IS', 'HERE', '\\xf0\\x9f\\x91\\x81', '\\xf0\\x9f\\x91\\x81', '\\xf0\\x9f\\x91\\x81', '\\xf0\\x9f\\x91\\x81', '\\xf0\\x9f\\x91\\x81', 'US', 'UK', 'Australia', 'Germany', 'Ireland', 'Italy', 'Netherlands', 'go', 'https', 'Amazon', "b'RT",

Upvotes: 0

Views: 2034

Answers (2)

alvas
alvas

Reputation: 122330

There's a hand-crafted tweet tokenizer in nltk:

>>> from nltk.tokenize import TweetTokenizer
>>> tt = TweetTokenizer()
>>> tweet = 'Thanks to the historic TAX CUTS that I signed into law, your paychecks are going way UP, your taxes are going way DOWN, and America is once again OPEN FOR BUSINESS! #FakeNews'
>>> tt.tokenize(tweet)
['Thanks', 'to', 'the', 'historic', 'TAX', 'CUTS', 'that', 'I', 'signed', 'into', 'law', ',', 'your', 'paychecks', 'are', 'going', 'way', 'UP', ',', 'your', 'taxes', 'are', 'going', 'way', 'DOWN', ',', 'and', 'America', 'is', 'once', 'again', 'OPEN', 'FOR', 'BUSINESS', '!', '#FakeNews']

Upvotes: 1

Richard Stoeffel
Richard Stoeffel

Reputation: 695

You can use a simple list comprehension if you have a specific list of words you are looking for, which would look like this:

words = word_tokenize(message.replace('\n',' '))
word_list = ['amazon', 'b']
filtered_words = [x for x in words if x in word_list]

If you are using python frequently you should delve into list comprehension, it will come up alot

Explanation of how list comprehension works?

http://www.pythonforbeginners.com/basics/list-comprehensions-in-python

Upvotes: 0

Related Questions