CODEWITHSUNDEEP
rloopsmatrixvectorizationapply
wolfsatthedoor
wolfsatthedoor

Reputation: 7303

Way to vectorize this loop? Multiply two matrices, store information, do this many times without looping

Suppose (small numbers in this example) I have an array that is 

3 x 14 x 5

call this 

set.seed(1)
dfarray=array(rnorm(5*3*14,0,1),dim=c(3,14,5))

I have a matrix that corresponds to this and is 

39 (which is 13*3) x 14

Call this matrix:

dfmat = matrix(rnorm(13*3*14,0,1),39,14)
dfmat = cbind(dfmat,rep(1:3,13))
dfmat = dfmat[order(dfmat [,15]),]
colnames(dfmat)[15]='unit'

What I want to do is run this loop:

 costs = c(0.45, 2.11, 1.05, 1.44, 0.88, 2.30, 1.96, 1.76, 2.06, 1.54, 1.69,1.75,0)
    p = c(1,2,3,1,4,3,2,1,4,1,3,4,0)
    profit=numeric(0)
    for(i in 1:3){
            j=13
            beta = dfarray[i,,]
            Xt = dfmat [which(dfmat [,'unit']==i),1:14]    #this takes a set of 13, Xt is 13x14

            Xbeta = exp( Xt %*% beta )
            iota = c(rep(1, j))
            denom = iota%*%Xbeta
            Prob =  (Xbeta/ (iota%*%denom))
            Eprob = rowSums(Prob)/5  #the 5 coming from the last dim of array
            profit = c(profit,sum((p-costs)*Eprob))

        }


     sum(profit)

I cannot think of a way to vectorize the part that the loop gets around by calling 

beta = dfarray[i,,]
Xt = dfmat [which(dfmat [,'unit']==i),]   #this takes a set of 13, Xt is 13x14

Upvotes: 4

Views: 213

Answers (1)

Khashaa
Khashaa

Reputation: 7373

To make my remarks in comment column clear, suppose we have dfmat as a list of matrices. It is almost always easier to work with a list of matrices than one big named matrix. Also if you want to fully vectorize the solution given here, you might want to obtain a block diagonal matrix using bdiag from Matrix package which acts on lists.

set.seed(1)
dfarray=array(rnorm(5*3*14,0,1),dim=c(3,14,5))
# dfmats as a list of matrices
dfmats <- lapply(1:3, function(i)matrix(rnorm(13*14), nrow=13))

Multiplication by iota is either colSums or rowSums, therefore we can simplify the operation as in f.

f <- function(Xbeta) rowSums(Xbeta / matrix(colSums(Xbeta), nrow=nrow(Xbeta), ncol=ncol(Xbeta), byrow=T)) / ncol(Xbeta)
#profits is written as a function for benchmarking
#cost and p are ignored as they can be easily added back in.
profits <- function(){ 
    Xbetas <- lapply(seq_len(dim(dfarray)[1]), function(i) exp(dfmats[[i]] %*% dfarray[i,,]))
    Eprobs <- lapply(Xbetas, f)
    unlist(Eprobs)
}

And your approach

profits1 <- function(){
    profit=numeric(0)
    for(i in 1:dim(dfarray)[1]){
        j=13
        beta = dfarray[i,,]
        Xt = dfmat [which(dfmat [,'unit']==i),1:14]    #this takes a set of 13, Xt is 13x14
        
        Xbeta = exp( Xt %*% beta )
        iota = c(rep(1, j))
        denom = iota%*%Xbeta
        deno <- colSums(Xbeta)
        s <- iota%*%denom
        Prob =  (Xbeta/ s)
        Eprob = rowSums(Prob)/dim(dfarray)[3]  #the 100 coming from the last dim of array
        profit = c(profit,Eprob)
        
    }
    return(profit)
}
dfmat <- do.call(rbind, dfmats)
dfmat <- cbind(dfmat,rep(1:3, each=13))
colnames(dfmat)[15]='unit'

Check that they give identical results

all.equal(profits(), profits1())
[1] TRUE

Benchmark

I ran this on AWS EC2 free-tie instance accessed via http://www.louisaslett.com/RStudio_AMI/.

dfarray=array(rnorm(100*10000*14,0,1),dim=c(10000,14,100))
dfmats <- lapply(1:10000, function(i)matrix(rnorm(13*14), nrow=13))

From your initial construct, you can transform dfmat to list dfmats as dfmats <- lapply(1:3, function(i)dfmat[which(dfmat [,'unit']==i),1:14]) but this is a very costly conversion. Creating dfmat from dfmats is reasonably less costly.

dfmat <- do.call(rbind, dfmats)
dfmat <- cbind(dfmat,rep(1:10000, each=13))
colnames(dfmat)[15]='unit'

Notice an exceptional speed-up using list, and perils of dreadful name-lookup cost.

system.time(a1 <- profits1())
#   user  system elapsed 
#250.885   4.442 255.394 
system.time(a <- profits())
#   user  system elapsed 
#  2.717   0.429   3.167 
all.equal(a, a1)
#[1] TRUE

PS: I've noticed that you have asked a several questions potentially related to this question, had all answered. I'll be happy if you share how you successfully put those in a good use.

Upvotes: 2

Related Questions