CODEWITHSUNDEEP
rparallel-processingcluster-computing
generic_user
generic_user

Reputation: 3562

R code parallelized using plyr and doMC: error message: Error in do.ply(i) : task 1 failed - "could not find function "getClass""

I'm just getting started learning how to use remote supercomputers for execution of parallelized code. I got a lot of initial help from this previous post, as well as one particularly helpful and patient XSEDE guy.

I'm only using one node (for the meantime), but each of its 32 cores. I'm using doMC instead of snow, because the guy at the supercomputer is in the process of getting Rmpi running. For now, 32 cores should be (more than) adequate for me. My script is of the sort:

define a bunch of functions
load the data
call libraries
require(doMC)
require(plyr)
registerDoMC(32)

main.function <- function(data){
    *the function*
    }

results = llply(1:500, function(idx){out<-main.function(data)},.parallel=TRUE)

save(results,file="trestles_results")

This runs fine on my own machine (setting it to run only a few times and registering only a couple of cores). But when I run it on the cluster, the output file shows that it ran each of the 500 iterations, but I get no output file, and I get the following error message:

Error in do.ply(i) : task 1 failed - "could not find function "getClass""
Calls: llply -> %dopar% -> <Anonymous>
Execution halted
Nodes:        trestles-10-28

Anybody have any idea what is going on here?

Thanks in advance!

Upvotes: 2

Views: 2958

Answers (2)

Dominik
Dominik

Reputation: 792

for completeness sake, *ply functions have a builtin .paropts where you can specify variables and packages to send to a cluster node

results = llply(1:500, 
          function(idx){out<-main.function(data)}, 
          .parallel=TRUE,
          .paropts=list(.export=c('data','main.function'),
                        .packages=.packages(all.available=T))

or you could be more selective with .packages, depending how complex main.function is. I think because you are calling main.function from inside the anonymous function you would need to pass main.function to the clusters too.

Upvotes: 2

Scott Kaiser
Scott Kaiser

Reputation: 317

One reason this can occur is that the environment hasn't been exported to the cores. I found one solution posted here, including sample code:

http://www.numbertheory.nl/2011/11/14/parallelization-using-plyr-loading-objects-and-packages-into-worker-nodes/

Upvotes: 0

Related Questions