CODEWITHSUNDEEP
rperformancefor-loop
useryk
useryk

Reputation: 309

Faster way to add new columns row-by-row onto a large dataframe

I have the following data frame.

> str(df)
'data.frame':   98444 obs. of  25 variables:
 $ count       : int  361 362 363 364 365 366 367 368 369 370 ...
 $ time        : num  3.01 3.02 3.02 3.03 3.04 ...
 $ H_Rx        : num  -164 -164 -164 -164 -164 ...
 $ H_Ry        : num  -10.7 -10.7 -10.7 -10.7 -10.7 ...
 $ H_Rz        : num  -174 -174 -174 -174 -174 ...
 $ H_Tx        : num  -0.00137 -0.00137 -0.00136 -0.00135 -0.00134 ...
 $ H_Ty        : num  1.67 1.67 1.67 1.67 1.67 ...
 $ H_Tz        : num  -0.194 -0.194 -0.194 -0.194 -0.194 ...
 $ C_Rx        : num  -13.4 -13.4 -13.5 -13.5 -13.6 ...
 $ C_Ry        : num  -14.7 -14.6 -14.5 -14.4 -14.4 ...
 $ C_Rz        : num  7.7 7.69 7.69 7.68 7.67 ...
 $ C_Tx        : num  0.00914 0.00914 0.00914 0.00914 0.00914 ...
 $ C_Ty        : num  1.21 1.21 1.21 1.21 1.21 ...
 $ C_Tz        : num  -0.0466 -0.0466 -0.0466 -0.0465 -0.0465 ...
 $ D_Rx        : num  -32.6 -32.6 -32.6 -32.6 -32.6 ...
 $ D_Ry        : num  -49 -49 -49 -49 -49 ...
 $ D_Rz        : num  1.91 1.91 1.91 1.92 1.92 ...
 $ D_Tx        : num  -0.0403 -0.0403 -0.0403 -0.0402 -0.0402 ...
 $ D_Ty        : num  1.63 1.63 1.63 1.63 1.63 ...
 $ D_Tz        : num  0.0214 0.0214 0.0214 0.0214 0.0215 ...
 $ part        : chr  "P2" "P2" "P2" "P2" ...
 $ freq        : chr  "100Hz" "100Hz" "100Hz" "100Hz" ...
 $ device      : chr  "A1" "A1" "A1" "A1" ...
 $ act         : chr  "Nod" "Nod" "Nod" "Nod" ...
 $ trial       : chr  "Rest" "Rest" "Rest" "Rest" ...
 - attr(*, "na.action")=Class 'omit'  Named int [1:133] 469 470 471 472 473 474 475 476 477 478 ...
  .. ..- attr(*, "names")= chr [1:133] "469" "470" "471" "472" ...

And I also have a list of matrices.

> str(listofmatrix)
List of 98444
 $ : num [1:4, 1] 0.0807 0.0165 -0.2062 1
 $ : num [1:4, 1] 0.0807 0.0165 -0.2062 1
  [list output truncated]

I extracted first three elements from each matrix in listofmatrix, placing them onto new columns in df, using a for-loop:

for (i in 1:nrow(df)) {
  df$D_Txnew[i] <- listofmatrix[[i]][1, 1]
  df$D_Tynew[i] <- listofmatrix[[i]][2, 1]
  df$D_Tznew[i] <- listofmatrix[[i]][3, 1]
}

It worked as intended, but the processing speed was less than desirable.

What are the different approaches to speed things up?

Upvotes: 2

Views: 49

Answers (1)

akrun
akrun

Reputation: 887511

Instead of assigning row by row, one option would be to extract the first elements from each of the matrices in 'listofmatrix' (as it have only a single column) to returns a list of vectors, rbind it and assign the output to new columns in 'df'.

df1[paste0("D_T", c("xnew", "ynew", "znew"))] <-  do.call(rbind, 
                     lapply(listofmatrix, `[`, 1:3))

By running the OP's code on 'df'

identical(df, df1)
#[1] TRUE

Benchmarks

Here are some benchmarks on a slightly bigger dataset

set.seed(142)
listofmatrix <- lapply(1:1e4, function(i) matrix(rnorm(4), ncol=1))
df <- data.frame(count = 1:1e4, act= sample(LETTERS, 1e4, replace=TRUE))
df1 <- df

system.time({
for (i in 1:nrow(df)) {
  df$D_Txnew[i] <- listofmatrix[[i]][1, 1]
  df$D_Tynew[i] <- listofmatrix[[i]][2, 1]
  df$D_Tznew[i] <- listofmatrix[[i]][3, 1]
}
})

#user  system elapsed 
#  1.94    0.00    1.94 
system.time({
df1[paste0("D_T", c("xnew", "ynew", "znew"))] <-  do.call(rbind, 
                     lapply(listofmatrix, `[`, 1:3))
})
# user  system elapsed 
#   0.02    0.00    0.02

data

set.seed(24)
listofmatrix <- lapply(1:5, function(i) matrix(rnorm(4), ncol=1))
df <- data.frame(count = 1:5, act= LETTERS[1:5]) 
df1 <- df

Upvotes: 2

Related Questions