CODEWITHSUNDEEP
rtext-miningtm
Luke Hansen
Luke Hansen

Reputation: 17

Cosine Similarity Matrix in R

I have a document term matrix, "mydtm" that I have created in R, using the 'tm' package. I am attempting to depict the similarities between each of the 557 documents contained within the dtm/corpus. I have been attempting to use a cosine similarity matrix using: mydtm_cosine <- dist(mydtm_matrix, method = "cosine", diag = F, upper = F) However the output matrix I get is huge with many missing values. Any help/suggestions would be much appreciated. Output Matrix

Upvotes: 0

Views: 1098

Answers (1)

CSJCampbell
CSJCampbell

Reputation: 2115

Likely you have few words which occur between your documents. You may wish to reduce the words in your term document matrix.

text <- c("term-document matrix is a mathematical matrix", 
    "we now have a tidy three-column",
    "cast into a Term-Document Matrix",
    "where the rows represent the text responses, or documents")
corpus <- VCorpus(VectorSource(text))
tdm <- TermDocumentMatrix(corpus, 
    control = list(wordLengths = c(1, Inf)))
occurrence <- apply(X = tdm, 
    MARGIN = 1, 
    FUN = function(x) sum(x > 0) / ncol(tdm))
occurrence
#            a          cast     documents          have 
#         0.75          0.25          0.25          0.25 
#         into            is  mathematical        matrix 
#         0.25          0.25          0.25          0.50 
#          now            or     represent    responses, 
#         0.25          0.25          0.25          0.25 
#         rows term-document          text           the 
#         0.25          0.50          0.25          0.25 
# three-column          tidy            we         where 
#         0.25          0.25          0.25          0.25 

quantile(occurrence, probs = c(0.5, 0.9, 0.99))
#    50%    90%    99% 
# 0.2500 0.5000 0.7025 

tdm_mat <- as.matrix(tdm[names(occurrence)[occurrence >= 0.5], ])
tdm_mat
#                Docs
# Terms           1 2 3 4
#   a             1 1 1 0
#   matrix        2 0 1 0
#   term-document 1 0 1 0

You can then calculate cosine similarity.

library(proxy)
dist(tdm_mat, method = "cosine", upper = TRUE)
#                       a    matrix term-document
# a                       0.2254033     0.1835034
# matrix        0.2254033               0.0513167
# term-document 0.1835034 0.0513167

Upvotes: 0

Related Questions