CODEWITHSUNDEEP
nlpbert-language-modeltransformer-modelsentence-similaritysentence-transformers
Tiago Bachiega de Almeida
Tiago Bachiega de Almeida

Reputation: 131

String comparison with BERT seems to ignore "not" in sentence

I implemented a string comparison method using SentenceTransformers and BERT like following

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

model = SentenceTransformer('sentence-transformers/all-distilroberta-v1')

sentences = [
    "I'm a good person",
    "I'm not a good person"
]

sentence_embeddings = model.encode(sentences)

cosine_similarity(
    [sentence_embeddings[0]],
    sentence_embeddings[1:]
)

Notice how my sentence examples are very similar but with the opposite meaning. The problem is the cosine similarity returns 0.9, indicating that these two strings are very similar in context when I expected it to return something closer to zero, as they have the opposite meanings.

How can I adapt my code to return a more accurate result?

Upvotes: 4

Views: 1390

Answers (3)

mithunpaul
mithunpaul

Reputation: 3536

Nothing surprising. This is probably a fall out of the bag of words based model. If you look at the both sentences they are exactly similar except for the not. This is where human intuition would rely on grammar and say, not negates the fact. However, BERT and other bag of words based models dont really learn it that way. Like the other answers here, retraining might help, but bottom line is if you want it to work off the shelf, you might want to consider some models that incorporate grammar like Quantum Natural Language Processing based models.

Upvotes: 0

David Dale
David Dale

Reputation: 11444

TL;DR: NLI is all you need

First, the cosine similarity is reasonably high, because the sentences are similar in the following sense:

  • They are about the same topic (evaluation of a person)
  • They are about the same subject ("I") and the same property ("being a good person")
  • They have similar syntactic structure
  • They have almost the same vocabulary

So, from the formal point of view, they should be considered similar. Moreover, from the practical point of view, they should often be considered similar. For example, if you google "GMO are causing cancer", you might find that the text with label "GMO are not causing cancer" is relevant.

Second, if you want to measure logical connection between sentences, cosine similarity of embeddings is just not expressive enough. This is because embeddings contain lots of semantic stylistic, lexical and syntactic information, but they are fixed-size (768-dimensional, in your case), so they cannot contain complete information about the meaning of both sentences. So you need another model with the following properties:

  1. It encodes both texts simultaneously, so it compares the texts themselves, not just their fixed-size embeddings
  2. It is explicitly trained to evaluate logical connection between sentences

The task of assesing logical connection between texts is called natural language inference (NLI), and its most common formulation is recognizing textual entailment (RTE): it is the problem of predicting whether the first sentence entails the second one.

There are lots of models trained for this task in the Huggingface repo, with roberta-large-mnli being a good one. You can use it to evaluate equivalence of two texts. If each text entails another, they are equivalent, so you can estimate the degree of equivalence as the product of the entailment scores in both directions.

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("roberta-large-mnli")
model = AutoModelForSequenceClassification.from_pretrained("roberta-large-mnli")

def test_entailment(text1, text2):
    batch = tokenizer(text1, text2, return_tensors='pt').to(model.device)
    with torch.no_grad():
        proba = torch.softmax(model(**batch).logits, -1)
    return proba.cpu().numpy()[0, model.config.label2id['ENTAILMENT']]

def test_equivalence(text1, text2):
    return test_entailment(text1, text2) * test_entailment(text2, text1)

print(test_equivalence("I'm a good person", "I'm not a good person"))  # 2.0751484e-07
print(test_equivalence("I'm a good person", "You are a good person"))  # 0.49342492
print(test_equivalence("I'm a good person", "I'm not a bad person"))   # 0.94236994

Upvotes: 8

Ashwin Geet D'Sa
Ashwin Geet D'Sa

Reputation: 7379

The results are not surprising. You have passed two sentences which are very similar, but have opposite meanings. The sentence embeddings are obtained from a model trained on generic corpora, hence, the embeddings given by the model are generally expected to close to each other if the sentences are similar. And that's what is happening, that the cosine similarity shows that the embedding are close to each other and so is the sentence. The sentences in the example may have opposite meanings, but they are similar to each other.

In case, if you expect two similar sentences with opposite meaning to be far away from each other, then you have to further fine-tune the model with kind of classification model (such as sentiment analysis, if your examples are based on positive and negative sentiments). or with some other relevant task.

Upvotes: 2

Related Questions