CODEWITHSUNDEEP
roptimizationparallel-processing
LeGeniusII
LeGeniusII

Reputation: 960

R optimParallel - could not find function

I'm trying to understand how optimParallel deals with embedded functions:

fn1 <- function(x){x^2-x+1}
fn2 <- function(x){fn1(x)}     # effectively fn1

cl <- parallel::makeCluster(parallel::detectCores()-1)
parallel::setDefaultCluster(cl = cl)

optimParallel::optimParallel(par = 0, fn = fn1)         # Worked
optimParallel::optimParallel(par = 0, fn = fn2)         # Not working

parallel::setDefaultCluster(cl=NULL)
parallel::stopCluster(cl)

Why the 2nd not working? the error message is 

Error in checkForRemoteErrors(val) : 
  3 nodes produced errors; first error: could not find function "fn1"

How to fix it?

Upvotes: 2

Views: 965

Answers (2)

Nairolf
Nairolf

Reputation: 2556

This problem is specific to the used cluster type:

If a FORK cluster is created after the function definitions, it works. FORK cluster are available on Linux like systems only:

library(optimParallel)
fn1 <- function(x) x^2-x+1 
fn2 <- function(x) fn1(x)
cl <- makeCluster(detectCores()-1, type="FORK")
setDefaultCluster(cl=cl)
optimParallel(par=0, fn=fn2)[[1]]
## [1] 0.5

For other cluster types one can add fn1 as an argument to fn2 and add it as ... argument to optimParallel():

fn1 <- function(x) x^2-x+1 
fn2 <- function(x, fn1) fn1(x)
cl <- makeCluster(detectCores()-1)
setDefaultCluster(cl=cl)
optimParallel(par=0, fn=fn2, fn1=fn1)[[1]]
## [1] 0.5

Alternatively, one can export fn1 to all R processes in the cluster:

fn1 <- function(x) x^2-x+1 
fn2 <- function(x) fn1(x)
cl <- makeCluster(detectCores()-1)
setDefaultCluster(cl=cl)
clusterExport(cl, "fn1")
optimParallel(par=0, fn=fn2)[[1]]
## [1] 0.5

Upvotes: 2

edsandorf
edsandorf

Reputation: 767

I believe the problem is that your function fn1 cannot be found on the cores. There is no reproducible example here for me to test, but I believe something along the these lines should work:

fn1 <- function(x){x^2-x+1}
fn2 <- function(x){fn1(x)}     # effectively fn1

cl <- parallel::makeCluster(parallel::detectCores()-1)
parallel::setDefaultCluster(cl = cl)

# Export the function to the cluster such that it exists in the global environment 
# of the worker
parallel::clusterExport(cl, "fn1")

optimParallel::optimParallel(par = 0, fn = fn2)         # Not working

parallel::setDefaultCluster(cl=NULL)
parallel::stopCluster(cl)

The key is that your function fn1 does not exist in the global environment of the worker.

Upvotes: 2

Related Questions