CODEWITHSUNDEEP
pythonspacy
rmf
rmf

Reputation: 45

Add new named entity to Spacy's en_core_web_sm model?

I'm following the example here on training a new entity type:

https://spacy.io/usage/training#example-new-entity-type

It works fine when I don't pass an existing model to it, and will correctly create a new model, which recognizes my new named entity e.g.

 python.exe train-new-entity-type.py

It also works fine when I pass an existing model to it (created by running it once before), and will correctly load the model in the dir/my_model dir, which still recognizes my new named entity e.g.

 python.exe train-new-entity-type.py -m dir/my_model

However, I want to train a new entity type, and add that to spacy's existing model, so that spacy will recognize it's own supported named entities as well as my new entity type, so I tried:

 python.exe train-new-entity-type.py -m en_core_web_sm

However, this didn't seem to work. Spacy's own supported named entities were recognized but they were not correct (vs just using the en_core_web_sm on its own without adding my new entity type to it), and my new entity type was no longer recognized at all.

Am I doing something wrong? Is this possible (adding named entities to en_core_web_sm)?

Upvotes: 1

Views: 1749

Answers (1)

aab
aab

Reputation: 11474

Read about the "catastrophic forgetting" problem when updating an existing model: https://spacy.io/usage/training#ner

It can be tricky to update an existing model, so it might be easier to train a separate model for your new entity type and add the NER component to the en_core_web_sm pipeline with a custom name. The main thing to watch out for is that you need to make sure the models are loaded with the same vocab so that you don't run into problems with the string store:

import spacy
nlp = spacy.load("en_core_web_sm")
custom_nlp = spacy.load("my_model", vocab=nlp.vocab)
nlp.add_pipe(custom_nlp.get_pipe("ner"), name="my_ner", before="ner")

Where you add it in the pipeline (before/after the existing ner) will determine which entity spans have priority, since the ner component won't modify existing entity spans.

Upvotes: 4

Related Questions