CODEWITHSUNDEEP
nlpstanford-nlp
Ziqi
Ziqi

Reputation: 2554

stanford-nlp NER from a list of tokens

Is there a way of using the stanford NER library to take input a list of tokens, and extract NEs?

I have checked the API but it is not explicit. Most of time the input is a String, a document, in both cases tokenization is done behind the scene.

In my case, I really have to do tokenization before and pass the list of tokens to the API. I have noticed that I can do:

List<HasWord> words = new ArrayList<>();

words.add(new Word("Tesco"));
..... //adding elements to words

List<CoreLabel> labels =classifier.classifySentence(words);

Is this correct?

Many thanks!!

Upvotes: 2

Views: 915

Answers (2)

StanfordNLPHelp
StanfordNLPHelp

Reputation: 8739

Here is one way to solve this issue:

import java.io.*;
import java.util.*;
import edu.stanford.nlp.io.*;
import edu.stanford.nlp.ling.*;
import edu.stanford.nlp.pipeline.*;
import edu.stanford.nlp.trees.*;
import edu.stanford.nlp.trees.TreeCoreAnnotations.*;
import edu.stanford.nlp.util.*;

public class NERPreToken {
    public static void main (String[] args) {
        Properties props = new Properties();
        props.setProperty("annotators",
            "tokenize, ssplit, pos, lemma, ner");
        props.setProperty("tokenize.whitespace", "true");
        StanfordCoreNLP pipeline = new StanfordCoreNLP(props);
        String [] tokensArray = {"Stephen","Colbert","hosts","a","show","on","CBS","."};
        List<String> tokensList = Arrays.asList(tokensArray);
        String docString = String.join(" ",tokensList);
        Annotation annotation = new Annotation(docString);
        pipeline.annotate(annotation);
        List<CoreMap> sentences = annotation.get(CoreAnnotations.SentencesAnnotation.class);
        for (CoreMap sentence : sentences) {
            List<CoreLabel> tokens = sentence.get(CoreAnnotations.TokensAnnotation.class);
            for (CoreLabel token : tokens) {
                System.out.println(token.word()+" "+token.get(CoreAnnotations.NamedEntityTagAnnotation.class));
            }
        }
    }
}

The key here is to start with your list of tokens and set the pipeline's property for tokenizing to just tokenize on white space. Then submit a String with your tokens joined by space.

Upvotes: 2

John Wiseman
John Wiseman

Reputation: 3137

You can use the Sentence.toCoreLabelList method:

String[] token_strs = {"John", "met", "Amy", "in", "Los", "Angeles"};
List<CoreLabel> tokens = edu.stanford.nlp.ling.Sentence.toCoreLabelList(token_strs);
for (CoreLabel cl : classifier.classifySentence(tokens)) {
  System.out.println(cl.toShorterString());
}

Output:

[Value=John Text=John Position=0 Answer=PERSON Shape=Xxxx DistSim=463]
[Value=met Text=met Position=1 Answer=O Shape=xxxk DistSim=476]
[Value=Amy Text=Amy Position=2 Answer=PERSON Shape=Xxx DistSim=396]
[Value=in Text=in Position=3 Answer=O Shape=xxk DistSim=510]
[Value=Los Text=Los Position=4 Answer=LOCATION Shape=Xxx DistSim=449]
[Value=Angeles Text=Angeles Position=5 Answer=LOCATION Shape=Xxxxx DistSim=199]

Upvotes: 2

Related Questions