CODEWITHSUNDEEP
pythonnlpnltk
bhat557
bhat557

Reputation: 105

How to calculate the distance in meaning of two words in Python

I am wondering if it's possible to calculate the distance/similarity between two related words in Python (like "fraud" and "steal"). These two words are not synonymous per se but they are clearly related. Are there any concepts/algorithms in NLP that can show this relationship numerically? Maybe via NLTK?

I'm not looking for the Levenshtein distance as that relates to the individual characters that make up a word. I'm looking for how the meaning relates.

Would appreciate any help provided.

Upvotes: 5

Views: 2938

Answers (3)

John Skiles Skinner
John Skiles Skinner

Reputation: 2028

As @jose_bacoy suggested above, the Gensim library can provide a measure of similarity between words using the word2vec technique. The below example is modified from this blog post. You can run it in Google Colab.

Google Colab comes with the Gensim package installed. We can import the part of it we require:

from gensim.models import KeyedVectors

We will download training data from Google News, and load it up

!wget -P /root/input/ -c "https://s3.amazonaws.com/dl4j-distribution/GoogleNews-vectors-negative300.bin.gz"
word_vectors = KeyedVectors.load_word2vec_format('/root/input/GoogleNews-vectors-negative300.bin.gz', binary=True)

This gives us a measure of similarity between any two words. To use your example:

word_vectors.similarity('fraud', 'steal')
>>> 0.19978741

Twenty percent similarity may be a surprisingly low level of similarity to you if you consider these words to be similar. But consider that fraud is a noun and steal is generally a verb. This will give them very different associations as viewed by word2vec.

They become much more similar if you modify the noun to become a verb:

word_vectors.similarity('defraud', 'steal')
>>> 0.43293646

Upvotes: 0

Hockenmaier
Hockenmaier

Reputation: 61

NLTK's wordnet is the tool you'd want to use for this. First get the set of all the senses of each word using:

synonymSet = wordnet.synsets(word)

Then loop through each possible sense of each of the 2 words and compare them to each other in a nested loop:

similarity = synonym1.res_similarity(synonym2,semcor_ic)

Either average that value or use the maximum you find; up to you.

This example is using a word similarity comparison that uses "IC" or information content. This will score similarity higher if the word is more specific, or contains more information, so generally it's closer to what we mean when we think about word similarity.

To use this stuff you'll need the imports and variables:

import nltk    
from nltk.corpus import wordnet
from nltk.corpus import wordnet_ic
semcor_ic = wordnet_ic.ic('ic-semcor.dat')

Upvotes: 0

Dlamini
Dlamini

Reputation: 285

My suggestion is as follows:

  • Put each word through the same thesaurus, to get a list of synonyms.
  • Get the size of the set of similar synonyms for the two words.
  • That is a measure of similarity between the words.

If you would like to do a more thorough analysis:

  • Also get the antonyms for each of the two words.
  • Get the size of the intersection of the sets of antonyms for the two words.

If you would like to go further!...

  • Put each word through the same thesaurus, to get a list of synonyms.
  • Use the top n (=5, or whatever) words from the query result to initiate a new query.
  • Repeat this to a depth you feel is adequate.
  • Make a collection of synonyms from the repeated synonym queries.
  • Get the size of the set of similar synonyms for the two words from the two collections of synonyms.
  • That is a measure of similarity between the words.

Upvotes: 3

Related Questions