CODEWITHSUNDEEP
spacyhuggingface-transformersspacy-3spacy-transformers
EnesZ
EnesZ

Reputation: 563

How to use Hugging Face transfomers with spaCy 3.0

Let's say that I want to include distilbert https://huggingface.co/distilbert-base-uncased from Hugging Face into spaCy 3.0 pipeline. I think that this is possible and I found some code on how to convert this model for spaCy 2.0 but it doesn't work in v3.0. What I really want is to load this model using something like this

nlp = spacy.load('path_to_distilbert')

Is it even possible and could you please provide the exact steps to do that.

Upvotes: 3

Views: 1608

Answers (2)

Sofie VL
Sofie VL

Reputation: 3106

You can use spacy-transformers to this end. In spaCy v3, you can train custom pipelines using a config file, where you would define the transformer component using any HF model you like in components.transformer.model.name:

[components.transformer]
factory = "transformer"
max_batch_items = 4096

[components.transformer.model]
@architectures = "spacy-transformers.TransformerModel.v1"
name = "bert-base-cased"
tokenizer_config = {"use_fast": true}

[components.transformer.model.get_spans]
@span_getters = "spacy-transformers.doc_spans.v1"

[components.transformer.set_extra_annotations]
@annotation_setters = "spacy-transformers.null_annotation_setter.v1"

You can then train any other component (NER, textcat, ...) on top of this pretrained transformer model, and the transformer weights will be further finetuned, too.

You can read more about this in the docs here: https://spacy.io/usage/embeddings-transformers#transformers-training

Upvotes: 3

David
David

Reputation: 2677

It appears that the only transformer that will work out of the box is their roberta-base model. In the docs it mentions being able to connect thousands of Huggingface models but there is no mention of how to add them to a SpaCy pipeline.

In the meantime if you wanted to use the roberta model you can do the following.

# install using spacy transformers
pip install spacy[transformers]
python -m spacy download en_core_web_trf

import spacy

nlp = spacy.load("en_core_web_trf")

Upvotes: 2

Related Questions