CODEWITHSUNDEEP
nlpmetricsmachine-translation
NobinPegasus
NobinPegasus

Reputation: 787

Why is both identical sentence's METEOR score used in Machine Translation not equals to one?

I have both identical sentences. But METEOR Score is not 1 it's 0.99.

Here's the code: score1 = nltk.translate.meteor_score.meteor_score(sentence1,sentence2)

Upvotes: 1

Views: 1035

Answers (1)

Jindřich
Jindřich

Reputation: 11240

It appears to be a sort of rounding error in computing the chunking penalty. It says into how many incoherent chunks the sentence must have been split to get the alignment between the reference and the hypothesis. Its purpose is to penalize hypothesis sentences that contain correct words and phrases, but in a very different order than in the reference.

See the paper introducing METEOR, Section 2:

enter image description here

If you get rid of the penalty by setting gamma to zero, you will always get 1.0 for identical sentence:

import nltk
sentence = "This is a very long sentence."

nltk.translate.meteor_score.meteor_score([sentence], sentence, gamma=1)
# returns 0.9953703703703703

nltk.translate.meteor_score.meteor_score([sentence], sentence, gamma=0)
# returns 1.0

Upvotes: 2

Related Questions