CODEWITHSUNDEEP
pythontf-idf
Economist_Ayahuasca
Economist_Ayahuasca

Reputation: 1642

cosine-similarity between consecutive pairs using whole articles in JSON file

I would like to calculate the cosine similarity for the consecutive pairs of articles in a JSON file. So far I manage to do it but.... I just realize that when transforming the tfidf of each article I am not using the terms from all articles available in the file but only those from each pair. Here is the code that I am using which provides the cosine-similarity coefficient of each consecutive pair of articles. 

import json
import nltk
with open('SDM_2015.json') as f:
    data = [json.loads(line) for line in f]

## Loading the packages needed:
import nltk, string
from sklearn.feature_extraction.text import TfidfVectorizer

## Defining our functions to filter the data

# Short for stemming each word (common root)
stemmer = nltk.stem.porter.PorterStemmer()

# Short for removing puctuations etc
remove_punctuation_map = dict((ord(char), None) for char in string.punctuation)

## First function that creates the tokens
def stem_tokens(tokens):
    return [stemmer.stem(item) for item in tokens]

## Function that incorporating the first function, converts all words into lower letters and removes puctuations maps (previously specified)
def normalize(text):
    return stem_tokens(nltk.word_tokenize(text.lower().translate(remove_punctuation_map)))

## Lastly, a super function is created that contains all the previous ones plus stopwords removal
vectorizer = TfidfVectorizer(tokenizer=normalize, stop_words='english')

## Calculation one by one of the cosine similatrity

def foo(x, y):
    tfidf = vectorizer.fit_transform([x, y])
    return ((tfidf * tfidf.T).A)[0,1]

my_funcs = {}
for i in range(len(data) - 1):
    x = data[i]['body']
    y = data[i+1]['body']
    foo.func_name = "cosine_sim%d" % i
    my_funcs["cosine_sim%d" % i] = foo
    print(foo(x,y))

Any idea of how to develop the cosine-similarity using the whole terms of all articles available in the JSON file rather than only those of each pair?

Kind regards,

Andres

Upvotes: 2

Views: 999

Answers (1)

flyingmeatball
flyingmeatball

Reputation: 7997

I think, based on our discussion above, you need to change the foo function and everything below. See the code below. Note that I haven't actually run this, since I don't have your data and no sample lines are provided.

## Loading the packages needed:
import nltk, string
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics import pairwise_distances
from scipy.spatial.distance import cosine
import json
from  sklearn.metrics.pairwise import cosine_similarity

with open('SDM_2015.json') as f:
    data = [json.loads(line) for line in f]

## Defining our functions to filter the data

# Short for stemming each word (common root)
stemmer = nltk.stem.porter.PorterStemmer()

# Short for removing puctuations etc
remove_punctuation_map = dict((ord(char), None) for char in string.punctuation)

## First function that creates the tokens
def stem_tokens(tokens):
    return [stemmer.stem(item) for item in tokens]

## Function that incorporating the first function, converts all words into lower letters and removes puctuations maps (previously specified)
def normalize(text):
    return stem_tokens(nltk.word_tokenize(text.lower().translate(remove_punctuation_map)))

## tfidf
vectorizer = TfidfVectorizer(tokenizer=normalize, stop_words='english')
tfidf_data = vectorizer.fit_transform(data)

#cosine dists
similarity matrix  = cosine_similarity(tfidf_data)

Upvotes: 1

Related Questions