CODEWITHSUNDEEP
algorithmtextlevenshtein-distance
Harris
Harris

Reputation: 1138

How to find similarity in texts

I have a database where users upload articles. I would like to make an algorithm where my web app will suggest similar texts according to the one the user reads.

I saw some examples like Levenshtein distance. But those algorithms measures distance for strings and not for whole articles. Is there a way to extract most significant keywords from text? Surely, I understand that "most significant" is an ambiguous term.

How do other sites manage this?

thanks a lot

Upvotes: 3

Views: 1140

Answers (2)

ptrk
ptrk

Reputation: 1830

You might try to correct the 'weight' of each word by the frequency it appears in all the articles. So the best indicators of similarity would be the words that appear only in the two compared ones and nowhere else. This would automatically disregard the common words (a, an, the, etc.) mentioned by @Gilbert Le Blanc.

Upvotes: 0

Gilbert Le Blanc
Gilbert Le Blanc

Reputation: 51445

Is there a way to extract most significant keywords from text?

Yes. Basically, you extract all the words from the text, sort the words by frequency, eliminate the common words (a, an, the, etc.) by matching them against a common word dictionary, and save the top 20 or more words, along with their frequency, from each article.

The number of top words you save is related to both the length of the article and the subject matter of all the articles. Less words work for general interest articles, while more words are necessary for special interest articles, like answers to programming questions.

Articles that match more than half of the top words could be considered related. The degree of relatedness would depend on the number of matching top words and the frequencies of the matching words.

You could calculate a relatedness score by multiplying the frequencies of each matched word from the two articles and summing all the products. The higher the score, the more the articles are related.

Upvotes: 1

Related Questions