CODEWITHSUNDEEP
rparallel-processingr-futurefurrr
Giovanni Colitti
Giovanni Colitti

Reputation: 2344

Running parLapply and future_map inside another function unnecessarily copies large objects to each worker

I was looking for an alternative to furrr:future_map() because when this function is run inside another function it copies all objects defined inside that function to each worker regardless of whether those objects are explicitly passed (https://github.com/DavisVaughan/furrr/issues/26).

It looks like parLapply() does the same thing when using clusterExport():

fun <- function(x) {
  big_obj <- 1
  cl <- parallel::makeCluster(2)
  parallel::clusterExport(cl, c("x"), envir = environment())
  parallel::parLapply(cl, c(1), function(x) {
    x + 1
    env <- environment()
    parent_env <- parent.env(env)
    return(list(this_env = env, parent_env = parent_env))
  })
}

res <- fun(1)
names(res[[1]]$parent_env)
#> [1] "cl"      "big_obj" "x"

Created on 2020-01-06 by the reprex package (v0.3.0)

How can I keep big_obj from getting copied to each worker? I am using a Windows machine so forking isn't an option.

Upvotes: 1

Views: 367

Answers (1)

F. Priv&#233;
F. Priv&#233;

Reputation: 11728

You can change the environment of your local function so that it does not include big_obj by assigning e.g. only the base environment.

fun <- function(x) {
  big_obj <- 1
  cl <- parallel::makeCluster(2)
  on.exit(parallel::stopCluster(cl), add = TRUE)
  parallel::clusterExport(cl, c("x"), envir = environment())
  local_fun <- function(x) {
    x + 1
    env <- environment()
    parent_env <- parent.env(env)
    return(list(this_env = env, parent_env = parent_env))
  }
  environment(local_fun) <- baseenv()
  parallel::parLapply(cl, c(1), local_fun)
}
res <- fun(1)
"big_obj" %in% names(res[[1]]$parent_env) # FALSE

Upvotes: 2

Related Questions