CODEWITHSUNDEEP
rparallel-processing
Zoe
Zoe

Reputation: 1000

Using parApply() inside a function

I am trying to parallelize some of my code and want to parallelize inside a function. Inside my function, the majority of the code is inside an apply(). How can I do this? I am working on windows and apparently can't use forking due to this reason. Here is an easy dummy example of what I want to achieve:

library(parallel)

# this works
no_cores <- detectCores()
clust <- makeCluster(no_cores)
var <- 1:5
clusterExport(clust, "var")
parSapply(clust, var, function(x) x^10)
stopCluster(clust)


# this does not work
parallel_exp <- function(input) {
  no_cores <- detectCores()
  clust <- makeCluster(no_cores)
  clusterExport(clust, "input")
  parSapply(clust, var, function(x) x^10)
  stopCluster(clust)
}
parallel_exp(1:5)

> Error in get(name, envir = envir) : object 'input' not found

As it seems, the cores can only access global variables if I understand this right? Can I circumvent this somehow?

Upvotes: 0

Views: 470

Answers (1)

HenrikB
HenrikB

Reputation: 6815

clusterExport() pulls from the global environment. Your input variable is not there, it's an argument local to the function, so you need to specify clusterExport(clust, "input", envir = environment()).

Upvotes: 3

Related Questions