CODEWITHSUNDEEP
rparallel-processing
Tim
Tim

Reputation: 7474

parallel lapply with random generation

What is the best way to translate the following function to parallel processing?

myapply <- function(n, FUN, ...) {
  lapply(1:n, function(i) { FUN(...) })
}

I need to run it with other functions used for random generation, e.g.

myapply(100, function(...) rnorm(100, ...), 1, 1)

so I need it to correctly handle random seeds. Moreover, I want the code to be portable and platform-independent, so parallel::mcapply does not work for me. I was considering foreach with doParallel and doRNG, but doRNG currently is not available for Windows on CRAN and was not updated for three years! I considered parallel::parLapply, but I have problems with properly doing clusterExport since I do not have a closed list of elements to export (I found this implementation, or here, but I hoped for something simpler).

Upvotes: 0

Views: 428

Answers (1)

HenrikB
HenrikB

Reputation: 6815

The future_lapply() function of the future package (I'm the author) provides parallel RNGs using L'Ecuyer-CMRG RNG streams, which is also what parallel::mclapply() uses. It works the same across all OSes and regardless of parallel backend.

The following works on all OSes, including Windows:

myapply <- function(n, FUN, ...) {
  future_lapply(1:n, function(i, ...) { 
    FUN(...)
  }, ..., future.seed = TRUE)
}

library("future")
plan(multiprocess)

y <- myapply(100, function(...) rnorm(100, ...), mean = 1, sd = 1)

Upvotes: 1

Related Questions