CODEWITHSUNDEEP
r
Chris
Chris

Reputation: 2256

Must speed up row-wise operations

I need to do row-wise operations more than 15 million times, but have too slow code. Here is a small reproducible example:

costMatrix1 <- rbind(c(4.2,3.6,2.1,2.3),c(9.6,5.5,7.2,4.9),c(2.6,8.2,6.4,8.3),c(4.8,3.3,6.8,5.7))
costMatrix2 <- costMatrix1 #Example, the costMatrix2 is actually different from costMatrix1

tbl_Filter <- rbind(c(0,0,0,4),c(1,2,3,4),c(1,0,3,0),c(1,2,0,0),c(1,2,0,4))

tbl_Sums <- data.frame(matrix(0, nrow=10, ncol=2))
colnames(tbl_Sums) <- c("Sum1","Sum2")

for (i in 1:nrow(tbl_Filter))
{
  tbl_Sums[i,1] <- sum(costMatrix1[tbl_Filter[i,],tbl_Filter[i,]])
  tbl_Sums[i,2] <- sum(costMatrix2[tbl_Filter[i,],tbl_Filter[i,]])
}

I think to replace the for-loop with ddply is the solution, but I can't get it to work.

Upvotes: 4

Views: 1423

Answers (4)

msp
msp

Reputation: 1069

In addition to the snowfall library mentioned above, there's also multicore that only implements the parallel version of lapply (called mclapply) and not of apply, but it's easy to rewrite the code to accommodate this:

newfun = function(n) {
  a <- sum(costMatrix1[tbl_Filter[n,],tbl_Filter[n,]])
  b <- sum(costMatrix2[tbl_Filter[n,],tbl_Filter[n,]])
  c(a,b)
}

nvec = matrix(data = 1:nrow(tbl_Filter), ncol = 1)

# single-core version using apply
out = t(apply(nvec,1,newfun))

# multicore version using mclapply
library(multicore)
out.list = mclapply(1:nrow(nvec),function(i)newfun(nvec[i,]))) 
out = do.call("rbind", out.list) 

# if the number of rows is huge, this will be much faster than do.call:
library(data.table)
out = rbindlist(out.list)

Upvotes: 0

Aaron - mostly inactive
Aaron - mostly inactive

Reputation: 37814

Not sure how the speed would compare, but you could also set up matrices to do matrix multiplication. This uses the fact that the information in your tbl_Filter has positive numbers in the columns you want to sum.

> ttt <- apply((tbl_Filter>0)*1,1,function(x) x %*% t(x))
> t(rbind(as.numeric(costMatrix1), as.numeric(costMatrix2)) %*% ttt)
     [,1]  [,2]
[1,]  5.7  11.4
[2,] 85.5 171.0
[3,] 15.3  30.6
[4,] 22.9  45.8
[5,] 43.9  87.8

Upvotes: 2

baha-kev
baha-kev

Reputation: 3059

If you have more than one CPU core, using snowfall might help you speed this up. The setup (pre-parallelization):

newfun = function(n) {
  a <- sum(costMatrix1[tbl_Filter[n,],tbl_Filter[n,]])
  b <- sum(costMatrix2[tbl_Filter[n,],tbl_Filter[n,]])
  c(a,b)
  }

nvec = matrix(data = 1:nrow(tbl_Filter), ncol = 1)

t = proc.time()
out = t(apply(nvec,1,function(x) newfun(x)))
proc.time() - t

Now, parallelized:

## load 'snowfall' package
require(snowfall)

## Initialize parallel operation --> choose number of CPUs here!
sfInit( parallel=TRUE, cpus=2 )

##################################################################
## 'Export' functions and variables to all "slaves" so that parallel calculations
## can occur

sfExport(list=list('newfun'))

sfExport('costMatrix1')
sfExport('costMatrix2')
sfExport('tbl_Filter')
sfExport('nvec')

## call function using sfApply; will return values as a list object
 out = sfApply(nvec, 1, function(x) newfun(x))

## stop parallel computing job
sfStop()

tbl_Sums = as.data.frame(t(out))
colnames(tbl_Sums) <- c("Sum1","Sum2")

Upvotes: 4

Andrie
Andrie

Reputation: 179558

If you have very large arrays to work with, you are probably better off sticking to base R.

Here is how you could use sapply to solve the summing problem for a single matrix. Then use it repeatedly on each input matrix:

sumOne <- function(cost, filter){
  sapply(1:nrow(filter), function(i)sum(cost[filter[i,], filter[i,]]))
}


cbind(
    sumOne(costMatrix1, tbl_Filter),
    sumOne(costMatrix2, tbl_Filter)
)

The results:

     [,1]  [,2]
[1,]  5.7  11.4
[2,] 85.5 171.0
[3,] 15.3  30.6
[4,] 22.9  45.8
[5,] 43.9  87.8

This should be much, much faster than your loop. Not because of the fact that a for loop is intrinsically slower than sapply (it's not), but because sapply automatically reserves memory for the result, combined with the fact that [<- is slow.

Upvotes: 5

Related Questions