CODEWITHSUNDEEP
javanlpnetbeans-7stanford-nlp
harshit
harshit

Reputation: 3846

Java Heap Space error on Stanford NER using Netbeans

I am using Stanford NER to parse a sentence for getting following tags : tokenize,ssplit,pos,lemma,ner. I also increased memory in netbeans by Project->Properties->Run->VM Options to -Xms1600M -Xmx1600M. Still I am getting Java out of memory exception. I am running 32-bit java, on windows 7 JDK version 1.7. Here is my code 

public ArrayList<String> NERTokensRet(String string) {
    ArrayList<String> myArr = new ArrayList<String>();


    props = new Properties();
    props.put("annotators", "tokenize,ssplit,pos,lemma,ner");
    pipeline = new StanfordCoreNLP(props);


    //     String resultString = string.replaceAll("[^\\p{L}\\p{N}]", " ");   
    Annotation annotation = new Annotation(string);
    pipeline.annotate(annotation);
    int j;
    for (CoreMap sentence : annotation.get(CoreAnnotations.SentencesAnnotation.class)) {
        List<CoreLabel> tokens = sentence.get(CoreAnnotations.TokensAnnotation.class);

        for (j = 0; j < tokens.size(); j++) {
            CoreLabel token = tokens.get(j);
            myArr.add("[" + token.originalText() + "," + token.tag() + "," + token.beginPosition() + "," + token.endPosition() + "]");
        }
        //System.out.println(myArr);
    }

    return myArr;
}

StackTrace : 

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:45)
at java.lang.StringBuilder.<init>(StringBuilder.java:68)
at java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:2998)
at java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:2819)
at java.io.ObjectInputStream.readString(ObjectInputStream.java:1598)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1319)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
at java.util.HashMap.readObject(HashMap.java:1030)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
at edu.stanford.nlp.ie.crf.CRFClassifier.loadClassifier(CRFClassifier.java:2255)
at edu.stanford.nlp.ie.AbstractSequenceClassifier.loadClassifier(AbstractSequenceClassifier.java:1444)
at edu.stanford.nlp.ie.AbstractSequenceClassifier.loadClassifier(AbstractSequenceClassifier.java:1421)
at edu.stanford.nlp.ie.AbstractSequenceClassifier.loadClassifier(AbstractSequenceClassifier.java:1500)
at edu.stanford.nlp.ie.AbstractSequenceClassifier.loadClassifier(AbstractSequenceClassifier.java:1487)
at edu.stanford.nlp.ie.crf.CRFClassifier.getClassifier(CRFClassifier.java:2386)
at edu.stanford.nlp.ie.ClassifierCombiner.loadClassifierFromPath(ClassifierCombiner.java:130)
at edu.stanford.nlp.ie.ClassifierCombiner.loadClassifiers(ClassifierCombiner.java:116)
at edu.stanford.nlp.ie.ClassifierCombiner.<init>(ClassifierCombiner.java:98)
at edu.stanford.nlp.ie.NERClassifierCombiner.<init>(NERClassifierCombiner.java:64)
at edu.stanford.nlp.pipeline.StanfordCoreNLP$6.create(StanfordCoreNLP.java:500)

Can some one please help as soon as possible?

Upvotes: 0

Views: 1587

Answers (2)

harshit
harshit

Reputation: 3846

I uninstalled everything(java and netbeans) and reinstalled everything(java and netbeans). It is still not able to assign -Xmx1400m but assigns -Xmx1000m and works well. Thank you everyone for your effort.

Upvotes: 0

Christopher Manning
Christopher Manning

Reputation: 9450

The stacktrace shows java running out of memory simply while loading the large models (features and weights) used for NER in CoreNLP. These do use a considerable amount of memory, but this is still very surprising. You don't say what OS, what JDK version, whether 32/64 bit, etc. you are using. But for your program above (with a main method added and a couple of types filled in), on Java 7u5 on Linux (CentOS 5), I can run it with -mx700m (with either 32 bit or 64 bit Java -- yay compressed oops). So I would think 1600m should be enough for any architecture/version.

So, I'd try:

  • Running with a bit more memory and seeing if that changes things (like -mx1800m)
  • If it doesn't, make sure that the VM really is getting the amount of memory you state above (even though what you write looks correct). E.g., try printing Runtime.getRuntime().maxMemory() / 1024 / 1024.

Upvotes: 2

Related Questions