How to calculate cosine similarity with already-calculated TFIDF scores

Question

I need to calculate cosine similarity between documents with already calculated TFIDF scores.

Usually I would use (e.g.) TFIDFVectorizer which would create a matrix of documents / terms, calculating TFIDF scores as it goes. I can't apply this because it will re-calculate TFIDF scores. This would be incorrect because the documents have already had a large amount of pre-processing including Bag of Words and IDF filtering (I will not explain why - too long).

Illustrative input CSV file:

Doc, Term,    TFIDF score
1,   apples,  0.3
1,   bananas, 0.7
2,   apples,  0.1
2,   pears,   0.9
3,   apples,  0.6
3,   bananas, 0.2
3,   pears,   0.2

I need to generate the matrix that would normally be generated by TFIDFVectorizer, e.g.:

  | apples | bananas | pears
1 | 0.3    | 0.7     | 0
2 | 0.1    | 0       | 0.9
3 | 0.6    | 0.2     | 0.2 

... so that I can calculate cosine similarity between documents.

I'm using Python 2.7 but suggestions for other solutions or tools are welcome. I can't easily switch to Python 3.

Edit:

This isn't really about transposing numpy arrays. It involves mapping TFIDF scores to a document / term matrix, with tokenized terms, and missing values filled in as 0.

Answer 1

If you can use pandas to read the whole csv file first in a dataframe, it becomes more easier.

import pandas as pd
import numpy as np
from sklearn.preprocessing import LabelEncoder

df = pd.read_csv('sample.csv', index_col=None, skipinitialspace=True)

# Converting the text Term to column index
le = LabelEncoder()
df['column']=le.fit_transform(df['Term'])

# Converting the Doc to row index
df['row']=df['Doc'] - 1

# Rows will be equal to max index of document
num_rows = max(df['row'])+1

# Columns will be equal to number of distinct terms
num_cols = len(le.classes_)

# Initialize the array with all zeroes
tfidf_arr = np.zeros((num_rows, num_cols))

# Iterate the dataframe and set the appropriate values in tfidf_arr
for index, row in df.iterrows():
    tfidf_arr[row['row'],row['column']]=row['TFIDF score']

Do go through comments and ask if not understand anything.

Answer 2

I suggest to use sparse matrices from scipy.sparse

from scipy.sparse import csr_matrix, coo_matrix
from sklearn.metrics.pairwise import cosine_similarity

input="""Doc, Term,    TFIDF score
1,   apples,  0.3
1,   bananas, 0.7
2,   apples,  0.1
2,   pears,   0.9
3,   apples,  0.6
3,   bananas, 0.2
3,   pears,   0.2"""

voc = {}

# sparse matrix representation: the coefficient
# with coordinates (rows[i], cols[i]) contains value data[i]
rows, cols, data = [], [], []

for line in input.split("\n")[1:]: # dismiss header

    doc, term, tfidf = line.replace(" ", "").split(",")

    rows.append(int(doc))

    # map each vocabulary item to an int
    if term not in voc:
        voc[term] = len(voc)

    cols.append(voc[term])
    data.append(float(tfidf))

doc_term_matrix = coo_matrix((data, (rows, cols)))

# compressed sparse row matrix (type of sparse matrix with fast row slicing)
sparse_row_matrix = doc_term_matrix.tocsr()

print("Sparse matrix")
print(sparse_row_matrix.toarray()) # convert to array

# compute similarity between each pair of documents
similarities = cosine_similarity(sparse_row_matrix)

print("Similarity matrix")
print(similarities)

Output:

Sparse matrix
[[0.  0.  0. ]
 [0.3 0.7 0. ]
 [0.1 0.  0.9]
 [0.6 0.2 0.2]]
Similarity matrix
[[0.         0.         0.         0.        ]
 [0.         1.         0.04350111 0.63344607]
 [0.         0.04350111 1.         0.39955629]
 [0.         0.63344607 0.39955629 1.        ]]

Answer 3

An inefficient hack that I will leave here in case it helps someone else. Other suggestions welcome.

def calculate_cosine_distance():
    unique_terms = get_unique_terms_as_list()

    tfidf_matrix = [[0 for i in range(len(unique_terms))] for j in range(TOTAL_NUMBER_OF_BOOKS)]

    with open(INPUT_FILE_PATH, mode='r') as infile:
        reader = csv.reader(infile.read().splitlines(), quoting=csv.QUOTE_NONE)

        # Ignore header row
        next(reader)

        for rows in reader:
            book = int(rows[0]) - 1 # To make it a zero-indexed array
            term_index = int(unique_terms.index(rows[1]))
            tfidf_matrix[book][term_index] = rows[2]

    # Calculate distance between book X and book Y
    print cosine_similarity(tfidf_matrix[0:1], tfidf_matrix)

def get_unique_terms_as_list():
    unique_terms = set()
    with open(INPUT_FILE_PATH, mode='rU') as infile:
        reader = csv.reader(infile.read().splitlines(), quoting=csv.QUOTE_NONE)
        # Skip header
        next(reader)
        for rows in reader:
            unique_terms.add(rows[1])

        unique_terms = list(unique_terms)
    return unique_terms