CODEWITHSUNDEEP
pythonregexnltk
user3182194
user3182194

Reputation: 1799

Python code to count frequent word pairs in nltk

I am confused how to find frequent word pairs in a file.I obtained the bigrams first but how to proceed from here? i tried stripping the punctuations using regexp before applying nltk.bigrams

raw=open("proj.txt","r").read()
tokens=nltk.word_tokenize(raw)
pairs=nltk.bigrams(tokens)
bigram_measures = nltk.collocations.BigramAssocMeasures()
trigram_measures = nltk.collocations.TrigramAssocMeasures()
finder = BigramCollocationFinder.from_words(pairs)
finder.apply_freq_filter(3)
finder.nbest(bigram_measures.pmi, 10)

Upvotes: 0

Views: 3116

Answers (2)

Justin O Barber
Justin O Barber

Reputation: 11591

You seem to call BigramCollocationFinder without importing it. The correct path is nltk.collocations.BigramCollocationFinder. So you can try this (be sure your text file has text!):

>>> import nltk
>>> raw = open('test2.txt').read()
>>> tokens = nltk.word_tokenize(raw)
# or, to exclude punctuation, use something like the following instead of the above line:
# >>> tokens = nltk.tokenize.RegexpTokenizer(r'\w+').tokenize(raw)
>>> pairs = nltk.bigrams(tokens)
>>> bigram_measures = nltk.collocations.BigramAssocMeasures()
>>> trigram_measures = nltk.collocations.TrigramAssocMeasures()
>>> finder = nltk.collocations.BigramCollocationFinder.from_words(pairs)  # note the difference here!
>>> finder.apply_freq_filter(3)
>>> finder.nbest(bigram_measures.pmi, 10)  # from the Old English text of Beowulf
[(('m\xe6g', 'Higelaces'), ('Higelaces', ',')), (('bearn', 'Ecg\xfeeowes'), ('Ecg\xfeeowes', ':')), (("''", 'Beowulf'), ('Beowulf', 'ma\xfeelode')), (('helm', 'Scyldinga'), ('Scyldinga', ':')), (('ne', 'cu\xfeon'), ('cu\xfeon', ',')), ((',', '\xe6r'), ('\xe6r', 'he')), ((',', 'helm'), ('helm', 'Scyldinga')), ((',', 'bearn'), ('bearn', 'Ecg\xfeeowes')), (('Ne', 'w\xe6s'), ('w\xe6s', '\xfe\xe6t')), (('Beowulf', 'ma\xfeelode'), ('ma\xfeelode', ','))]

Upvotes: 1

gfritz
gfritz

Reputation: 579

It sounds like you just want the list of word pairs. If so, I think you mean to use finder.score_ngrams like so: 

bigram_measures = nltk.collocations.BigramAssocMeasures()
finder = BigramCollocationFinder.from_words(tokens)
scores = finder.score_ngrams( bigram_measures.raw_freq )
print scores

There are other scoring metrics that can be used. It sounds like you want just the frequency, but other scoring metrics for general Ngrams are here - http://nltk.googlecode.com/svn-/trunk/doc/api/nltk.metrics.association.NgramAssocMeasures-class.html

Upvotes: 0

Related Questions