CODEWITHSUNDEEP
nlpspacyentity-linking
IVR
IVR

Reputation: 1858

How to evaluate SciSpaCy's entity linking

I'm using SciSpaCy's Entity Linker with a custom knowledge base. As I'm updating some components of my application (e.g. the underlying language model, sentence tokenization pipeline, the knowledge base itself, etc), I'm noticing that (1) the number of entities that the application picks up changes and (2) the linked concepts themselves change (not the detected entities but the concepts that are linked to these entities). With this in mind, I'd like to be able to evaluate my entity-linking application.

Unfortunately, I cannot seem to find any resources for that. I was hoping to find either an evaluation library of some sort (assuming we are not just interested in a confusion matrix) or a "gold standard" dataset with entities in various forms (e.g. abbreviated, inflected, etc) and the expected linked concept.

I'm afraid I'm a novice in this field which is why I'm reaching out here, hoping that anyone might be able to point me to a set of useful resources or share some tips with me.

Many thanks in advance.

Upvotes: 3

Views: 416

Answers (1)

user27502
user27502

Reputation: 1

Original code used in spacy to evaluate entity linker & compute metrics

https://github.com/explosion/projects/blob/master/nel-wikipedia/entity_linker_evaluation.py

Also See

  1. https://github.com/ad-freiburg/elevant
  2. https://github.com/wikilinks/neleval
  3. https://towardsdatascience.com/building-a-biomedical-entity-linker-with-llms-d385cb85c15a shows computation on own data, gives code , analysis.
  4. https://github.com/sebastianruder/NLP-progress/blob/master/english/entity_linking.md

Upvotes: 0

Related Questions