CODEWITHSUNDEEP
algorithm
Andrew Bullock
Andrew Bullock

Reputation: 37378

Algorithm to compare similarity of English sentences

I have a collection of sentences, and I need to analyse them to see how similar they are.

Are there any established algorithms to do this?

I care about:

I've used Levenshtein distance and n-grams for spelling before, although I'm not entirely confident if these translate to my purposes.

Naively, "I don't care about spelling differences, typos can be treated as different words" although perhaps it would be nice to account for this.

perhaps some hybrid of splitting the sentence at spaces and one of the above (or other) algorithms would be a starting point

What options are available? Any advice?

Thanks!

Upvotes: 11

Views: 3108

Answers (2)

Matt
Matt

Reputation: 1198

To ignore inflections you should look into stemming algorithms: http://en.wikipedia.org/wiki/Porter_stemmer

They reduce words to their root forms.

Upvotes: 0

Szabolcs
Szabolcs

Reputation: 25703

This paper compares several sentence similarity measures. Perhaps you can use one of them as is, or modify it for your needs.

Otherwise sentence similarity measure is a good key term to google for.

Upvotes: 13

Related Questions