CODEWITHSUNDEEP
javanlpstanford-nlppos-tagger
Satyanarayana Krishnamurthy
Satyanarayana Krishnamurthy

Reputation: 11

A surprising tag from Stanford-POS-Tagger

I used Stanford-POS-Tagger on the following text (from a Times of India News Report on the player-auctions under Indian Premier League):

Royal Challengers Bangalore are used to making strong statements at the Indian Premier League auctions and they did so again on Saturday (February 6) with the marquee signing of seasoned Australian all-rounder Shane Watson. The staggering Rs 9.5 crore that the team paid for the 34-year-old made him the costliest buy this year.

The Vijay Mallya-owned side fought off stiff competition from new entrants Rising Pune Supergiants and defending champions Mumbai Indians to snare the former Rajasthan Royals star. Watson, a battling right-handed batsman and handy medium-pacer, will add serious bite to the Virat Kohli-led Bengaluru side still chasing their maiden title.

For the last sentence, in the II-para, the Stanford-POS-Tagger tagged the first word 'Watson' as a base verb! I searched Chambers' Twentieth Century Dictionary to see if the word 'watson' is a verb, but I could locate no such entry!

I got the following output from some functions I have ran in my code:

Watson,VB aDT battlingVBG rightJJ handedNN batsmanNN andCC handyJJ mediumNN pacer,NN willMD addVB seriousJJ biteNN toTO theDT ViratNNP KohliNNP ledVBD BengaluruNNP sideNN stillRB chasingVBG theirPRP$ maidenJJ title.NN

Upvotes: 1

Views: 631

Answers (3)

Michail Michailidis
Michail Michailidis

Reputation: 12181

using maven packages I can't replicate this behavior. Make sure you are using the correct tagger. I am using the default one:

<dependency>
   <groupId>edu.stanford.nlp</groupId>
   <artifactId>stanford-parser</artifactId>
   <version>3.6.0</version>
</dependency>
<dependency>
    <groupId>edu.stanford.nlp</groupId>
    <artifactId>stanford-corenlp</artifactId>
    <version>3.6.0</version>
</dependency>
<dependency>
    <groupId>edu.stanford.nlp</groupId>
    <artifactId>stanford-corenlp</artifactId>
    <version>3.6.0</version>
    <classifier>models</classifier>
</dependency> 

public void test(){
    MaxentTagger tagger = new MaxentTagger(MaxentTagger.DEFAULT_JAR_PATH);
    String test = "Royal Challengers Bangalore are used to making strong statements at the Indian Premier League auctions and they did so again on Saturday (February 6) with the marquee signing of seasoned Australian all-rounder Shane Watson. The staggering Rs 9.5 crore that the team paid for the 34-year-old made him the costliest buy this year.The Vijay Mallya-owned side fought off stiff competition from new entrants Rising Pune Supergiants and defending champions Mumbai Indians to snare the former Rajasthan Royals star. Watson, a battling right-handed batsman and handy medium-pacer, will add serious bite to the Virat Kohli-led Bengaluru side still chasing their maiden title.";
    String taggedString = tagger.tagString(test);
    System.out.println(taggedString);
}

This code returns

Royal_NNP Challengers_NNS Bangalore_NNP are_VBP used_VBN to_TO making_VBG strong_JJ statements_NNS at_IN the_DT Indian_JJ Premier_NNP League_NNP auctions_NNS and_CC they_PRP did_VBD so_RB again_RB on_IN Saturday_NNP -LRB-_-LRB- February_NNP 6_CD -RRB-_-RRB- with_IN the_DT marquee_JJ signing_NN of_IN seasoned_JJ Australian_JJ all-rounder_NN Shane_NNP Watson_NNP ._. The_DT staggering_JJ Rs_NN 9.5_CD crore_VBP that_IN the_DT team_NN paid_VBN for_IN the_DT 34-year-old_JJ made_VBD him_PRP the_DT costliest_JJS buy_VB this_DT year.The_NNP Vijay_NNP Mallya-owned_JJ side_NN fought_VBD off_RP stiff_JJ competition_NN from_IN new_JJ entrants_NNS Rising_VBG Pune_NNP Supergiants_NNPS and_CC defending_VBG champions_NNS Mumbai_NNP Indians_NNPS to_TO snare_VB the_DT former_JJ Rajasthan_NNP Royals_NNPS star_NN ._. Watson_NNP ,_, a_DT battling_VBG right-handed_JJ batsman_NN and_CC handy_JJ medium-pacer_NN ,_, will_MD add_VB serious_JJ bite_NN to_TO the_DT Virat_NNP Kohli-led_NNP Bengaluru_NNP side_NN still_RB chasing_VBG their_PRP$ maiden_JJ title_NN ._.

Upvotes: 0

alvas
alvas

Reputation: 121992

The problem seems to be that you have not tokenized your text before POS tagging.

As @ChristopherManning showed, the output of the Stanford POS tagger will be correct if you tokenized your text before you tag.

Using the CoreNLP on command line:

alvas@ubi:~/stanford-corenlp-full-2015-12-09$ echo """Royal Challengers Bangalore are used to making strong statements at the Indian Premier League auctions and they did so again on Saturday (February 6) with the marquee signing of seasoned Australian all-rounder Shane Watson. The staggering Rs 9.5 crore that the team paid for the 34-year-old made him the costliest buy this year.

The Vijay Mallya-owned side fought off stiff competition from new entrants Rising Pune Supergiants and defending champions Mumbai Indians to snare the former Rajasthan Royals star. Watson, a battling right-handed batsman and handy medium-pacer, will add serious bite to the Virat Kohli-led Bengaluru side still chasing their maiden title.""" > watson.txt
alvas@ubi:~/stanford-corenlp-full-2015-12-09$ 
alvas@ubi:~/stanford-corenlp-full-2015-12-09$ java -cp "*" -Xmx2g edu.stanford.nlp.pipeline.StanfordCoreNLP -annotators tokenize,ssplit,pos -outputFormat json -file watson.txt
[main] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator tokenize
[main] INFO edu.stanford.nlp.pipeline.TokenizerAnnotator - TokenizerAnnotator: No tokenizer type provided. Defaulting to PTBTokenizer.
[main] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator ssplit
[main] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator pos
Reading POS tagger model from edu/stanford/nlp/models/pos-tagger/english-left3words/english-left3words-distsim.tagger ... done [1.6 sec].

Processing file /home/alvas/stanford-corenlp-full-2015-12-09/watson.txt ... writing to /home/alvas/stanford-corenlp-full-2015-12-09/watson.txt.json
Annotating file /home/alvas/stanford-corenlp-full-2015-12-09/watson.txt
done.
Annotation pipeline timing information:
TokenizerAnnotator: 0.1 sec.
WordsToSentencesAnnotator: 0.0 sec.
POSTaggerAnnotator: 0.1 sec.
TOTAL: 0.1 sec. for 110 tokens at 791.4 tokens/sec.
Pipeline setup: 1.6 sec.
Total time for StanfordCoreNLP pipeline: 1.9 sec

The output will be saved in watson.txt.json and with some munging:

>>> import json
>>> with open('watson.txt.json') as fin:
...     output = json.load(fin)
... 
>>> for sent in output['sentences']:
...     print ' '.join([tok['word']+'/'+tok['pos'] for tok in sent['tokens']]) + '\n'
... 

Royal/NNP Challengers/NNS Bangalore/NNP are/VBP used/VBN to/TO making/VBG strong/JJ statements/NNS at/IN the/DT Indian/JJ Premier/NNP League/NNP auctions/NNS and/CC they/PRP did/VBD so/RB again/RB on/IN Saturday/NNP -LRB-/-LRB- February/NNP 6/CD -RRB-/-RRB- with/IN the/DT marquee/JJ 

signing/NN of/IN seasoned/JJ Australian/JJ all-rounder/NN Shane/NNP Watson/NNP ./.

The/DT staggering/JJ Rs/NN 9.5/CD crore/VBP that/IN the/DT team/NN paid/VBN for/IN the/DT 34-year-old/JJ made/VBD him/PRP the/DT costliest/JJS buy/VB this/DT year/NN ./.

The/DT Vijay/NNP Mallya-owned/JJ side/NN fought/VBD off/RP stiff/JJ competition/NN from/IN new/JJ entrants/NNS Rising/VBG Pune/NNP Supergiants/NNPS and/CC defending/VBG champions/NNS Mumbai/NNP Indians/NNPS to/TO snare/VB the/DT former/JJ Rajasthan/NNP Royals/NNPS star/NN ./.

Watson/NNP ,/, a/DT battling/VBG right-handed/JJ batsman/NN and/CC handy/JJ medium-pacer/NN ,/, will/MD add/VB serious/JJ bite/NN to/TO the/DT Virat/NNP Kohli-led/NNP Bengaluru/NNP side/NN still/RB chasing/VBG their/PRP$ maiden/JJ title/NN ./.

Note that if you're using the Stanford CoreNLP on command line it will NOT allow you to POS tag without tokenization:

alvas@ubi:~/stanford-corenlp-full-2015-12-09$ java -cp "*" -Xmx2g edu.stanford.nlp.pipeline.StanfordCoreNLP -annotators pos -outputFormat json -file watson.txt[main] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator pos
Reading POS tagger model from edu/stanford/nlp/models/pos-tagger/english-left3words/english-left3words-distsim.tagger ... done [1.4 sec].
Exception in thread "main" java.lang.IllegalArgumentException: annotator "pos" requires annotator "tokenize"
    at edu.stanford.nlp.pipeline.StanfordCoreNLP.construct(StanfordCoreNLP.java:375)
    at edu.stanford.nlp.pipeline.StanfordCoreNLP.<init>(StanfordCoreNLP.java:139)
    at edu.stanford.nlp.pipeline.StanfordCoreNLP.<init>(StanfordCoreNLP.java:135)
    at edu.stanford.nlp.pipeline.StanfordCoreNLP.main(StanfordCoreNLP.java:1214)

Regardless of whether you're using Stanford POS tagger through the GUI, command-line, a python API or directly through importing the library in your Java code, it is advisable to sentence tokenize your text, then word tokenize each sentence before POS tagging them.

The Stanford CoreNLP API provides an example of how you can annotate your data in Java: http://stanfordnlp.github.io/CoreNLP/api.html

Upvotes: 4

Christopher Manning
Christopher Manning

Reputation: 9450

I couldn't reproduce this tagging with either the default model using the current version (3.6.0) or the slower, better model.... (But in general, the tagger isn't limited to a dictionary of taggings and can choose the tag it thinks most appropriate in context.)

$ java -cp "*" edu.stanford.nlp.tagger.maxent.MaxentTagger -model edu/stanford/nlp/models/pos-tagger/english-left3words/english-left3words-distsim.tagger -textFile watson.txt 

Loading default properties from tagger edu/stanford/nlp/models/pos-tagger/english-left3words/english-left3words-distsim.tagger

Reading POS tagger model from edu/stanford/nlp/models/pos-tagger/english-left3words/english-left3words-distsim.tagger ... done [0.7 sec].

Royal_NNP Challengers_NNS Bangalore_NNP are_VBP used_VBN to_TO making_VBG strong_JJ statements_NNS at_IN the_DT Indian_JJ Premier_NNP League_NNP auctions_NNS and_CC they_PRP did_VBD so_RB again_RB on_IN Saturday_NNP -LRB-_-LRB- February_NNP 6_CD -RRB-_-RRB- with_IN the_DT marquee_JJ signing_NN of_IN seasoned_JJ Australian_JJ all-rounder_NN Shane_NNP Watson_NNP ._.
The_DT staggering_JJ Rs_NN 9.5_CD crore_VBP that_IN the_DT team_NN paid_VBN for_IN the_DT 34-year-old_JJ made_VBD him_PRP the_DT costliest_JJS buy_VB this_DT year_NN ._.
The_DT Vijay_NNP Mallya-owned_JJ side_NN fought_VBD off_RP stiff_JJ competition_NN from_IN new_JJ entrants_NNS Rising_VBG Pune_NNP Supergiants_NNPS and_CC defending_VBG champions_NNS Mumbai_NNP Indians_NNPS to_TO snare_VB the_DT former_JJ Rajasthan_NNP Royals_NNPS star_NN ._.
Watson_NNP ,_, a_DT battling_VBG right-handed_JJ batsman_NN and_CC handy_JJ medium-pacer_NN ,_, will_MD add_VB serious_JJ bite_NN to_TO the_DT Virat_NNP Kohli-led_NNP Bengaluru_NNP side_NN still_RB chasing_VBG their_PRP$ maiden_JJ title_NN ._.

Tagged 110 words at 859.38 words per second.

Upvotes: 1

Related Questions