CODEWITHSUNDEEP
javanlpdeep-learningdeeplearning4j
Exi-torrere
Exi-torrere

Reputation: 21

Problems with loading nlp model with deeplearning4j

so I'm trying to implement the free deeplearning library for Java called deeplearning4j to tackle a classification task in nlp.

public static void Learn(String labelledDataFileName) throws Exception {
    ParagraphVectors paragraphVectors = new ParagraphVectors();
    InMemoryLookupCache cache = new InMemoryLookupCache();
    LabelleDataIterator iterator = new LabelleDataIterator(new File(labelledDataFileName));
    TokenizerFactory t = new DefaultTokenizerFactory();
    t.setTokenPreProcessor(new CommonPreprocessor());
    paragraphVectors = new ParagraphVectors.Builder()
            .minWordFrequency(1)
            .iterations(3)
            .learningRate(0.025)
            .minLearningRate(0.001)
            .layerSize(400)
            .batchSize(1000)
            .epochs(1)
            .iterate(iterator)
            .trainWordVectors(true)
            .vocabCache(cache)
            .tokenizerFactory(t)
            .build();
    paragraphVectors.fit();
    WordVectorSerializer.writeFullModel(paragraphVectors, MODEL_FILE_NAME);
}

pretty standard, not much different from the sample provided on the net. The trained model after being fitted is then saved to a text file with the method writeFullModel. Then it can be loaded with this method

WordVectorSerializer.loadFullModel(MODEL_FILE_NAME);

The problem is, it doesn't seem to work when the model gets big. For a model file of size 120Mb, I keep getting this 

Exception in thread "main" java.lang.IllegalArgumentException: Illegal slice 7151
at org.nd4j.linalg.api.ndarray.BaseNDArray.slice(BaseNDArray.java:2852)
at org.nd4j.linalg.api.ndarray.BaseNDArray.tensorAlongDimension(BaseNDArray.java:753)
at org.nd4j.linalg.api.ndarray.BaseNDArray.vectorAlongDimension(BaseNDArray.java:830)
at org.nd4j.linalg.api.ndarray.BaseNDArray.getRow(BaseNDArray.java:3628)
at org.deeplearning4j.models.embeddings.loader.WordVectorSerializer.loadFullModel(WordVectorSerializer.java:523)

It loasd fine with a small model file though. Any help would be appreciated, thank you so much.

Upvotes: 2

Views: 525

Answers (1)

user6384481
user6384481

Reputation:

When the IllegalArgumentException is thrown, you must check the call stack in Java’s stack trace and locate the method that produced the wrong argument.

Here is the BaseNDArray.java class and see the lines or search for the methods you've got the error in! see this method:

public INDArray slice(int slice) {
    int slices = slices();
    if(slice >= slices)
        throw new IllegalArgumentException("Illegal slice " + slice);

so,for the 120MB file, slice >= slices! see if this can help and reply how did you solve the problem!

Upvotes: 2

Related Questions