CODEWITHSUNDEEP
python-3.xdeep-learningembeddinghuggingface-transformerstransformer-model
MAC
MAC

Reputation: 1515

How to generate sentence embedding using long-former model

I am using Hugging Face mrm8488/longformer-base-4096-finetuned-squadv2 pre-trained model https://huggingface.co/mrm8488/longformer-base-4096-finetuned-squadv2.

I want to generate sentence level embedding. I have a data-frame which has a text column.

I am using this code:

import torch
from transformers import AutoTokenizer, AutoModelForQuestionAnswering
ckpt = "mrm8488/longformer-base-4096-finetuned-squadv2"
tokenizer = AutoTokenizer.from_pretrained(ckpt)
model = AutoModelForQuestionAnswering.from_pretrained(ckpt)

text = "Huggingface has democratized NLP. Huge thanks to Huggingface for this." # I will pas text-column here from my data-frame
#question = "What has Huggingface done ?"
encoding = tokenizer(question, text, return_tensors="pt")
# I don't want to use it for Question-Answer use-case. I just need the sentence embeddings
input_ids = encoding["input_ids"]

# default is local attention everywhere
# the forward method will automatically set global attention on question tokens
attention_mask = encoding["attention_mask"]

How can I do modification in the above code to generate embedding for sentences. ?

I have the following examples:

                           Text
i've added notes to the claim and it's been escalated for final review
after submitting the request you'll receive an email confirming the open request.
hello my name is person and i'll be assisting you
this is sam and i'll be assisting you for date.
I'll return the amount as asap.
ill return it to you.

Upvotes: 2

Views: 2286

Answers (1)

cronoik
cronoik

Reputation: 19365

The Longformer uses a local attention mechanism and you need to pass a global attention mask to let one token attend to all tokens of your sequence.

import torch
from transformers import LongformerTokenizer, LongformerModel
ckpt = "mrm8488/longformer-base-4096-finetuned-squadv2"
tokenizer = LongformerTokenizer.from_pretrained(ckpt)
model = LongformerModel.from_pretrained(ckpt)

text = "Huggingface has democratized NLP. Huge thanks to Huggingface for this." # I will pas text-column here from my data-frame
#question = "What has Huggingface done ?"
encoding = tokenizer(text, return_tensors="pt")

global_attention_mask = [1].extend([0]*encoding["input_ids"].shape[-1])

encoding["global_attention_mask"] = global_attention_mask

# I don't want to use it for Question-Answer use-case. I just need the sentence embeddings
# default is local attention everywhere
# the forward method will automatically set global attention on question tokens

o = model(**encoding)

sentence_embedding = o.last_hidden_state[:,0]

You should keep in mind that mrm8488/longformer-base-4096-finetuned-squadv2 was not pre-trained to produce meaningful sentence embeddings and faces the same issues as the MLM pre-trained BERT's regarding sentence embeddings.

Upvotes: 2

Related Questions