CODEWITHSUNDEEP
pythonnlpnltkstanford-nlp
cpd1
cpd1

Reputation: 789

Stanford NLP Tagger via NLTK - tag_sents splits everything into chars

I'm hoping someone has experience with this as I'm unable to find any comments online besides a bug report from 2015 regarding the NERtagger which is probably the same.

Anyway, I'm trying to batch process text to get around the poor performing base tagger. From what I understand, tag_sents should help.

from nltk.tag.stanford import StanfordPOSTagger
from nltk import word_tokenize
import nltk

stanford_model = 'stanford-postagger/models/english-bidirectional-distsim.tagger'
stanford_jar = 'stanford-postagger/stanford-postagger.jar'
tagger = StanfordPOSTagger(stanford_model, stanford_jar)
tagger.java_options = '-mx4096m'
text = "The quick brown fox jumps over the lazy dog."
print tagger.tag_sents(text)

Except no matter what I pass to the tag_sents method, the text gets split up into chars instead of words. Anyone know why it doesn't work properly? This works as expected...

tag(text)

I tried splitting the sentence into tokens as well to see if that helped but same treatment

Upvotes: 2

Views: 1479

Answers (2)

alvas
alvas

Reputation: 122122

The tag_sents function takes a list of list of strings.

tagger.tag_sents(word_tokenize("The quick brown fox jumps over the lazy dog."))

Here's a useful idiom:

 tagger.tag_sents(word_tokenize(sent) for sent in sent_tokenize(text))

where text is a string.

Upvotes: 3

Antimony
Antimony

Reputation: 2240

Another variation of what alvas said, which worked for me: tagger.tag_sents([[text]]).

Upvotes: 0

Related Questions