CODEWITHSUNDEEP
scalaapache-sparknlpapache-spark-mllibjohnsnowlabs-spark-nlp
Marsellus Wallace
Marsellus Wallace

Reputation: 18601

How to load a spark-nlp pre-trained model from disk

From the spark-nlp Github page I downloaded a .zip file containing a pre-trained NerCRFModel. The zip contains three folders: embeddings, fields, and metadata.

How do I load that into a Scala NerCrfModel so that I can use it? Do I have to drop it into HDFS or the host where I launch my Spark Shell? How do I reference it?

Upvotes: 4

Views: 3286

Answers (1)

AlbertoAndreotti
AlbertoAndreotti

Reputation: 510

you just need to provide the path where the folders you mentioned are contained,

import com.johnsnowlabs.nlp.annotators.ner.crf.NerCrfModel
val path = "path/to/unziped/file/folder"
val model = NerCrfModel.read.load(path)
// use your model
model.setInputCols(someCol)
model.transform(yourData) // which contains 'someCol',

As long as I remember, you can place the folder in local FS or distributed FS, hope this helps other users as well!.

best, Alberto.

Upvotes: 5

Related Questions