CODEWITHSUNDEEP
pythonnltk
JayGatsby
JayGatsby

Reputation: 1621

finding bigrams in unicode text with nltk

I'm trying to find most common bigrams in a unicode text. Here is the code which I'm using:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
from __future__ import unicode_literals
import nltk
from nltk.collocations import *
import codecs
line = ""
open_file = codecs.open('s.txt', 'r', encoding='utf-8').read()
for val in open_file:
    line += val.lower()
tokens = line.split()

bigram_measures = nltk.collocations.BigramAssocMeasures()
finder = BigramCollocationFinder.from_words(tokens)
finder.apply_freq_filter(1)
a = finder.ngram_fd.viewitems()
for i,j in a:
    print i,j

s.txt file includes this text: çalışmak naber çösd bfkd

Here is the output:

(u'\xe7\xf6sd', u'bfkd') 1
(u'naber', u'\xe7\xf6sd') 1
(u'\xe7al\u0131\u015fmak', u'naber') 1

But I want it in this format:

çalışmak naber 1
naber çösd 1
çösd bfkd 1

How can I solve this unicode problem?

Upvotes: 0

Views: 1246

Answers (1)

erip
erip

Reputation: 16935

You need to print the elements of the tuple explicitly, not the entire tuple.

#!/usr/bin/env python
# -*- coding: utf-8 -*-
from __future__ import unicode_literals
import nltk
from nltk.collocations import *
import codecs
line = ""
open_file = codecs.open('s.txt', 'r', encoding='utf-8').read()
for val in open_file:
    line += val.lower()
tokens = line.split()

bigram_measures = nltk.collocations.BigramAssocMeasures()
finder = BigramCollocationFinder.from_words(tokens)
finder.apply_freq_filter(1)
a = finder.ngram_fd.viewitems()
for i, j in a:
  print("{0} {1} {2}".format(i[0], i[1], j))

test.py

l = [((u'\xe7\xf6sd', u'bfkd'), 1), ((u'naber', u'\xe7\xf6sd'), 1), ((u'\xe7al\u0131\u015fmak', u'naber'), 1)]
for i, j in l:
  print("{0} {1} {2}".format(i[0], i[1], j))

Running:

14:58 $ python test.py
çösd bfkd 1
naber çösd 1
çalışmak naber 1

Upvotes: 2

Related Questions