CODEWITHSUNDEEP
pythonmachine-learningspacynamed-entity-recognition
Navya
Navya

Reputation: 49

How to calculate the overall accuracy of custom trained spacy ner model with confusion matrix?

I'm trying to evaluate my custom trained Spacy NER model. How to find the overall accuracy with confusion matrix for the model.

I tried evaluating the model with spacy scorer which gives precision, recall and token accuracy with the below reference,

Evaluation in a Spacy NER model

I expect the output in confusion matrix instead of individual precision, recall and token accuracy.

Upvotes: 4

Views: 3922

Answers (1)

Jeffrey Hiraki
Jeffrey Hiraki

Reputation: 173

Here is a good read for creating Confusion Matrices for Spacy NER models. It is based on the BILOU format used by Spacy. It is good for small portions of text but when bigger documents are evaluated a Confusion Matrix is hard to read because most pieces of the text are O-labeled.

What you can do is create two lists, one with predicted values per word and one with the true values per word and compare those using the sklearn.metrics.confusion_matrix() function.

from sklearn.metrics import confusion_matrix
y_true = [O,O,O,B-PER,I-PER]
y_pred = [O,O,O,B-PER,O]
confusion_matrix(y_true, y_pred, labels=["O", "B-PER", "I-PER"])

You can also use the plot_confusion_matrix() function from the same library to get a visual output, however this requires scikit-learn 0.23.1 or above and is only usable with scikit-learn classifiers.

As written in this stackoverflow question, this is a way to use the confusion_matrix() from scikit-learn without their plot.

from sklearn.metrics import confusion_matrix

labels = ['business', 'health']
cm = confusion_matrix(y_test, pred, labels)
print(cm)
fig = plt.figure()
ax = fig.add_subplot(111)
cax = ax.matshow(cm)
plt.title('Confusion matrix of the classifier')
fig.colorbar(cax)
ax.set_xticklabels([''] + labels)
ax.set_yticklabels([''] + labels)
plt.xlabel('Predicted')
plt.ylabel('True')
plt.show()

Upvotes: 1

Related Questions