CODEWITHSUNDEEP
nlpnltkn-gram
Hugo Cipriano
Hugo Cipriano

Reputation: 31

N-gram ignoring line-breaks

I am relatively new with nltk but recently I have had a problem and I can't seem to find a solution for it.

I have a text like this

Monkeys like bananas.

The sky is blue.

I am trying to do bigrams and by using word_tokenizer it creates these bigrams:

Monkeys like 1 like bananas 1 bananas The 1 The sky 1 sky is 1 is blue 1

How can I make it work so it doesn't create the bigram

bananas The 1 ?

import nltk 
from nltk.util import ngrams
corpus = open("my.txt").read()
tokens = nltk.word_tokenize(corpus)
bigrams = ngrams(tokens,2)
print Counter(bigrams)

Upvotes: 2

Views: 295

Answers (1)

Ahmed Masud
Ahmed Masud

Reputation: 22402

TLDR;

use sent_tokenize before using bigrams. (Make sure punkt is installed by using nltk.download('punkt');

Then:

>>> x = nltk.sent_tokenize("The Monkeys like bananas. The sky is blue")
['The Monkeys like bananas.', 'The sky is blue.']

>>> for x in a:
...     list(nltk.bigrams(nltk.word_tokenize(x)))
...
[('The', 'Monkeys'), ('Monkeys', 'like'), ('like', 'bananas'), ('bananas', '.')]
[('The', 'sky'), ('sky', 'is'), ('is', 'blue')]

Details

1) You're missing the point of bigrams.

They are used to train the engine to identify words that are collocated.

So you have to parse the content a little bit before you use bigrams. Thus the use of sent_tokenize followed by word_tokenize before using bigrams in the example above. You probably want to remove punctuation as well by preprocessing the text through a proper tokenizer.

Upvotes: 1

Related Questions