CODEWITHSUNDEEP
pythonnltk
Are you Shure
Are you Shure

Reputation: 85

NLTK tokenize questions

Just a quick beginners question here with NLTK. I am trying to tokenize and generate bigrams, trigrams and quadgrams from a corpus.

I need to add <s> to the beginning of my lists and </s> to the end in place of a period if there is one.

The list is taken from the brown corpus in nltk. (and a specific section at that)

so.. I have

from nltk.corpus import brown
news = brown.sents(categories = 'editorial')

Am I making this too difficult? Thanks a lot.

Upvotes: 1

Views: 415

Answers (1)

unutbu
unutbu

Reputation: 879701

import nltk.corpus as corpus

def mark_sentence(row):
    if row[-1] == '.':
        row[-1] = '</s>'
    else:
        row.append('</s>')
    return ['<s>'] + row

news = corpus.brown.sents(categories = 'editorial')
for row in news[:5]:
    print(mark_sentence(row))

yields

['<s>', 'Assembly', 'session', 'brought', 'much', 'good', '</s>']
['<s>', 'The', 'General', 'Assembly', ',', 'which', 'adjourns', 'today', ',', 'has', 'performed', 'in', 'an', 'atmosphere', 'of', 'crisis', 'and', 'struggle', 'from', 'the', 'day', 'it', 'convened', '</s>']
['<s>', 'It', 'was', 'faced', 'immediately', 'with', 'a', 'showdown', 'on', 'the', 'schools', ',', 'an', 'issue', 'which', 'was', 'met', 'squarely', 'in', 'conjunction', 'with', 'the', 'governor', 'with', 'a', 'decision', 'not', 'to', 'risk', 'abandoning', 'public', 'education', '</s>']
['<s>', 'There', 'followed', 'the', 'historic', 'appropriations', 'and', 'budget', 'fight', ',', 'in', 'which', 'the', 'General', 'Assembly', 'decided', 'to', 'tackle', 'executive', 'powers', '</s>']
['<s>', 'The', 'final', 'decision', 'went', 'to', 'the', 'executive', 'but', 'a', 'way', 'has', 'been', 'opened', 'for', 'strengthening', 'budgeting', 'procedures', 'and', 'to', 'provide', 'legislators', 'information', 'they', 'need', '</s>']

Upvotes: 3

Related Questions