group by and calculate the cosince similarity of text data from year t-1 to t?

Question

I have some text data in a similar format to the below data.

I am trying to group by ID and years and then calculate the cosince similarity between the text and see how much the document has changed from t-1 to t.

So for ID1 take the year 2002 and compare it to the document in 2001. Then take 2001 and compare it to the document in 2000.

Text      ID     years

Text 1    ID1     2000
Text 2    ID1     2001
Text 3    ID1     2002
Text 4    ID2     2000
Text 5    ID2     2001
...

What I am doing at the moment is simply comparing the documents witin a given year but I would like to also compare them from the previous year.

library(tidytext)
library(textTinyR)
library(tm)
library(dplyr)

df <- data %>%
  filter(years == "2000") %>%
  select(ID, text) %>%
  setNames(c("doc_id", "text"))

docs = VCorpus(DataframeSource(df))
dtm <- DocumentTermMatrix(docs)

dtm <- DocumentTermMatrix(docs, control = list(weighting = function(x)
  weightTfIdf(x, normalize = FALSE)))

inspect(dtm)
findFreqTerms(dtm, 2)

m <- as.matrix(dtm)

library(Matrix)
m <- Matrix(m, sparse = TRUE)

d <- dist(m)

plot(d)


############


my.df <- as.data.frame(inspect(dtm))
my.df.scale <- scale(my.df)
d <- dist(my.df.scale,method="euclidean")
plot(d)

Data:

text <- c("Football is a family of team sports that involve, to varying degrees, kicking a ball to score a goal. Unqualified, the word football is understood to refer to whichever form of football is the most popular in the regional context in which the word appears. Sports commonly called football in certain places include association football (known as soccer in some countries); gridiron football (specifically American football or Canadian football); Australian rules football; rugby football (either rugby league or rugby union); and Gaelic football.[1][2] These different variations of football are known as football codes.",
          "Football is a family of team sports that involve, to varying degrees, kicking a ball to score a goal. Unqualified, the word football is understood to refer to whichever form of football is the most popular in the regional context in which the word appears. Sports commonly called football in certain places include association football (known as soccer in some countries); gridiron football (specifically American football or Canadian football); Australian rules football; rugby football (either rugby league or rugby union); and Gaelic football.[1][2]",
          "Tennis is a racket sport that can be played individually against a single opponent (singles) or between two teams of two players each (doubles). Each player uses a tennis racket that is strung with cord to strike a hollow rubber ball covered with felt over or around a net and into the opponent's court. The object of the game is to maneuver the ball in such a way that the opponent is not able to play a valid return. The player who is unable to return the ball will not gain a point, while the opposite player will.",
          "Tennis is a racket sport that can be played individually against a single opponent (singles) or between two teams of two players each (doubles). Each player uses a tennis racket that is strung with cord to strike a hollow rubber ball covered with felt over or around a net and into the opponent's court. The object of the game is to maneuver the ball in such a way that the opponent is not able to play a valid return.",
          "Rugby refers to the team sports rugby league and rugby union. Legend claims that rugby football was started about 1845 in Rugby School, Rugby, Warwickshire, England, although forms of football in which the ball was carried and tossed date to medieval times. Rugby eventually split into two sports in 1895 when twenty-one clubs split from the original Rugby Football Union, to form the Northern Union (later to be named rugby league in 1922) in the George Hotel, Huddersfield, Northern England over the issue of payment to players, thus making rugby league the first code to turn professional and pay its players, rugby union turned fully professional in 1995. Both sports are run by their respective world governing bodies World Rugby (rugby union) and the Rugby League International Federation (rugby league). Rugby football was one of many versions of football played at English public schools in the 19th century.[1][2] Although rugby league initially used rugby union rules, they are now wholly separate sports. In addition to these two codes, both American and Canadian football evolved from rugby football.",
          "Badminton is a racquet sport played using racquets to hit a shuttlecock across a net. Although it may be played with larger teams, the most common forms of the game are singles (with one player per side) and doubles (with two players per side). Badminton is often played as a casual outdoor activity in a yard or on a beach; formal games are played on a rectangular indoor court. Points are scored by striking the shuttlecock with the racquet and landing it within the opposing side's half of the court.",
          "Cricket is a bat-and-ball game played between two teams of eleven players on a field at the centre of which is a 20-metre (22-yard) pitch with a wicket at each end, each comprising two bails balanced on three stumps. The batting side scores runs by striking the ball bowled at the wicket with the bat, while the bowling and fielding side tries to prevent this and dismiss each player (so they are out). Means of dismissal include being bowled, when the ball hits the stumps and dislodges the bails, and by the fielding side catching the ball after it is hit by the bat, but before it hits the ground. When ten players have been dismissed, the innings ends and the teams swap roles. The game is adjudicated by two umpires, aided by a third umpire and match referee in international matches. They communicate with two off-field scorers who record the match's statistical information.",
          "Basketball is a team sport in which two teams, most commonly of five players each, opposing one another on a rectangular court, compete with the primary objective of shooting a basketball (approximately 9.4 inches (24 cm) in diameter) through the defender's hoop (a basket 18 inches (46 cm) in diameter mounted 10 feet (3.048 m) high to a backboard at each end of the court) while preventing the opposing team from shooting through their own hoop. A field goal is worth two points, unless made from behind the three-point line, when it is worth three. After a foul, timed play stops and the player fouled or designated to shoot a technical foul is given one or more one-point free throws. The team with the most points at the end of the game wins, but if regulation play expires with the score tied, an additional period of play (overtime) is mandated.",
          "Hockey is a sport in which two teams play against each other by trying to manoeuvre a ball or a puck into the opponent's goal using a hockey stick. There are many types of hockey such as bandy, field hockey, and ice hockey. In most of the world, hockey refers to field hockey, while in Canada, the United States, Finland, Sweden, Latvia, the Czech Republic and Slovakia, hockey usually refers to ice hockey.[1]")


ID <- c("ID1", "ID1", "ID1", "ID2", "ID2", "ID2", "ID3", "ID3", "ID3")

years <- c("2000", "2001", "2002", "2000", "2001", "2002", "2000", "2001", "2002")

data <- data.frame(text, ID, years)

EDIT:

This is what I currently have (slightly updated data)…

text <- c("Football is a family of team sports that involve, to varying degrees, kicking a ball to score a goal. Unqualified, the word football is understood to refer to whichever form of football is the most popular in the regional context in which the word appears. Sports commonly called football in certain places include association football (known as soccer in some countries); gridiron football (specifically American football or Canadian football); Australian rules football; rugby football (either rugby league or rugby union); and Gaelic football.[1][2] These different variations of football are known as football codes.",
          "Football is a family of team sports that involve, to varying degrees, kicking a ball to score a goal. Unqualified, the word football is understood to refer to whichever form of football is the most popular in the regional context in which the word appears. Sports commonly called football in certain places include association football (known as soccer in some countries); gridiron football (specifically American football or Canadian football); Australian rules football; rugby football (either rugby league or rugby union); and Gaelic football.[1][2]",
          "Tennis is a racket sport that can be played individually against a single opponent (singles) or between two teams of two players each (doubles). Each player uses a tennis racket that is strung with cord to strike a hollow rubber ball covered with felt over or around a net and into the opponent's court. The object of the game is to maneuver the ball in such a way that the opponent is not able to play a valid return. The player who is unable to return the ball will not gain a point, while the opposite player will.",
          "Tennis is a racket sport that can be played individually against a single opponent (singles) or between two teams of two players each (doubles). Each player uses a tennis racket that is strung with cord to strike a hollow rubber ball covered with felt over or around a net and into the opponent's court. The object of the game is to maneuver the ball in such a way that the opponent is not able to play a valid return.",
          "Rugby refers to the team sports rugby league and rugby union. Legend claims that rugby football was started about 1845 in Rugby School, Rugby, Warwickshire, England, although forms of football in which the ball was carried and tossed date to medieval times. Rugby eventually split into two sports in 1895 when twenty-one clubs split from the original Rugby Football Union, to form the Northern Union (later to be named rugby league in 1922) in the George Hotel, Huddersfield, Northern England over the issue of payment to players, thus making rugby league the first code to turn professional and pay its players, rugby union turned fully professional in 1995. Both sports are run by their respective world governing bodies World Rugby (rugby union) and the Rugby League International Federation (rugby league). Rugby football was one of many versions of football played at English public schools in the 19th century.[1][2] Although rugby league initially used rugby union rules, they are now wholly separate sports. In addition to these two codes, both American and Canadian football evolved from rugby football.",
          "Badminton is a racquet sport played using racquets to hit a shuttlecock across a net. Although it may be played with larger teams, the most common forms of the game are singles (with one player per side) and doubles (with two players per side). Badminton is often played as a casual outdoor activity in a yard or on a beach; formal games are played on a rectangular indoor court. Points are scored by striking the shuttlecock with the racquet and landing it within the opposing side's half of the court.",
          "Cricket is a bat-and-ball game played between two teams of eleven players on a field at the centre of which is a 20-metre (22-yard) pitch with a wicket at each end, each comprising two bails balanced on three stumps. The batting side scores runs by striking the ball bowled at the wicket with the bat, while the bowling and fielding side tries to prevent this and dismiss each player (so they are out). Means of dismissal include being bowled, when the ball hits the stumps and dislodges the bails, and by the fielding side catching the ball after it is hit by the bat, but before it hits the ground. When ten players have been dismissed, the innings ends and the teams swap roles. The game is adjudicated by two umpires, aided by a third umpire and match referee in international matches. They communicate with two off-field scorers who record the match's statistical information.",
          "Basketball is a team sport in which two teams, most commonly of five players each, opposing one another on a rectangular court, compete with the primary objective of shooting a basketball (approximately 9.4 inches (24 cm) in diameter) through the defender's hoop (a basket 18 inches (46 cm) in diameter mounted 10 feet (3.048 m) high to a backboard at each end of the court) while preventing the opposing team from shooting through their own hoop. A field goal is worth two points, unless made from behind the three-point line, when it is worth three. After a foul, timed play stops and the player fouled or designated to shoot a technical foul is given one or more one-point free throws. The team with the most points at the end of the game wins, but if regulation play expires with the score tied, an additional period of play (overtime) is mandated.",
          "Football is a family of team sports that involve, to varying degrees, kicking a ball to score a goal. Unqualified, the word football is understood to refer to whichever form of football is the most popular in the regional context in which the word appears. Sports commonly called football in certain places include association football (known as soccer in some countries); gridiron football (specifically American football or Canadian football); Australian rules football; rugby football (either rugby league or rugby union); and Gaelic football.[1][2] These different variations of football are known as football codes.")


ID <- c("ID1", "ID1", "ID1", "ID1", "ID1", "ID3", "ID3", "ID3", "ID3")

years <- c("2000", "2001", "2002", "2003", "2004", "2000", "2001", "2002", "2003")

data <- data.frame(text, ID, years)


data2 <- data %>%
  mutate(doc_id=paste(ID, years, sep="_")) %>% #combines id and year
  group_split(ID) #splits df

data2 <- lapply(data2, function(x) x[c(-2,-3)])#removes extra cols
data2 <- lapply(data2, as.data.frame)#changes dfs in list to dataframes

data3 <- plyr::ldply(data2)

docs = VCorpus(DataframeSource(data3))
dtm <- DocumentTermMatrix(docs)
dtm <- DocumentTermMatrix(docs, control = list(weighting = function(x)
  weightTfIdf(x, normalize = FALSE)))



inspect(dtm)

dist(dtm)

Output:

          ID1_2000  ID1_2001  ID1_2002  ID1_2003  ID1_2004  ID3_2000  ID3_2001  ID3_2002
ID1_2001  5.247895                                                                      
ID1_2002 24.106356 22.578237                                                            
ID1_2003 21.983718 20.296410  9.234505                                                  
ID1_2004 39.816574 39.430571 44.036195 42.911001                                        
ID3_2000 28.303690 27.093798 28.283461 26.546848 46.153218                              
ID3_2001 34.775122 33.755114 34.161337 32.969805 50.375486 37.634369                    
ID3_2002 34.972794 33.937542 34.864501 33.546066 50.386962 37.526433 42.609499          
ID3_2003  0.000000  5.247895 24.106356 21.983718 39.816574 28.303690 34.775122 34.972794

I think this is getting closer to what I want, here though I am not really interested in computing the distance between ID3, 2003 and ID1, 2000. So I think I just want to keep the diagonal elements...

EDIT 2:

I also think that now the tf-idf is wrong and biased… I should calculate the tf-idf for 2000 using the two texts…

Football is a family of team sports that involve,...

Badminton is a racquet sport played using racquets...

Whereas at the momento its taking all text across all years for the caluclation.

Answer 1

Not sure helpful or am understanding correct but can split the data by id and filter by year, wrap the body in a function and pass the list to function. Something vaguely like this maybe:

data2<-data %>% filter(years != "2000") %>% #or change to 2002
  mutate(doc_id=paste(ID, years, sep="_")) %>% #combines id and year
  group_split(ID) #splits df
data2 <- lapply(data2, function(x) x[c(-2,-3)])#removes extra cols
data2 <- lapply(data2, as.data.frame)#changes dfs in list to dataframes


gofun<-function(df){

  docs = VCorpus(DataframeSource(df))
dtm <- DocumentTermMatrix(docs)
 ......
}

EDIT:

I *think if you want the diagonals the docs have to be split in advance and then the Vcorpus made. I found this subsequence function here.

data <- data.frame(text, ID, years) 

#creates subsequence --(still learning R so I think there is a different way to do this)
get_seq=function(df,n){
  res=c()
  for(i in 1:(nrow(df)-n+1)){
    res=rbind(res,cbind(df[i:(i+n-1),],"seq_label"=i))
  }
  res
}

data<-data %>% 
  group_by(ID) %>%
  do(data.frame(mod=get_seq(.,2)))%>% ungroup() %>% 
  select(ID, text=mod.text, years=mod.years, seq=mod.seq_label)


data2 <- data %>%
  mutate(doc_id=paste(ID, years, sep="_")) %>% #combines id and year
  group_split(ID,seq) #splits 


data2 <- lapply(data2, function(x) x[c(-1,-3,-4)])#removes extra cols
data2 <- lapply(data2, as.data.frame)#changes dfs in list to dataframes



gofun<-function(df){
docs = VCorpus(DataframeSource(df))
dtm <- DocumentTermMatrix(docs)
dtm <- DocumentTermMatrix(docs, control = list(weighting = function(x)
  weightTfIdf(x, normalize = FALSE)))

inspect(dtm)

dist(dtm)

}

lapply(data2, gofun)
[[1]]
         ID1_2000
ID1_2001  2.44949

[[2]]
         ID1_2001
ID1_2002 14.24781

[[3]]
         ID1_2002
ID1_2003        3

[[4]]
         ID1_2003
ID1_2004 22.69361

[[5]]
         ID3_2000
ID3_2001 13.37909

[[6]]
         ID3_2001
ID3_2002 15.16575

[[7]]
         ID3_2002
ID3_2003 17.08801