CODEWITHSUNDEEP
rparallel-processingmpi
Antoine Prince
Antoine Prince

Reputation: 31

Rmpi mpi.remote.exec() can't access user-defined function

I am trying to execute a simple function using the Rmpi package in R. When I run the mpi.remote.exec() function using a user-defined function inside, the program doesn't find my function.

library(Rmpi)
mpi.spawn.Rslaves()

df1 <- data.frame("x" = 1:2, "y" = 1:2)
df2 <- data.frame("x" = 2:3, "y" = 2:3)
df3 <- data.frame("x" = 3:4, "y" = 3:4)
df4 <- data.frame("x" = 4:5, "y" = 4:5)

fun <- function(x) {
  input <- get(paste0("df", x), envir = .GlobalEnv) 
  return(input)
}

array <- 1:4
MPIresults <- mpi.remote.exec(cmd = fun, array)

Doing so, I get the following error message:

$slave1
[1] "Error in fun(1:4) : could not find function \"fun\"\n"
attr(,"class")
[1] "try-error"
attr(,"condition")
<simpleError in fun(1:4): could not find function "fun">

Any idea on what I'm missing here?

Thanks!

Upvotes: 2

Views: 171

Answers (1)

A_Murphy
A_Murphy

Reputation: 314

Sorry if this is not suitable but I think I can help with the error. There are two problems I can see here.

First is that you have to explicitly pass the custom function to the slaves, the function mpi.bcast.Robj2slave will do this for you. See the documentation for a more detailed explanation.

So the code will run with this:

library(Rmpi)
mpi.spawn.Rslaves()

df1 <- data.frame("x" = 1:2, "y" = 1:2)
df2 <- data.frame("x" = 2:3, "y" = 2:3)
df3 <- data.frame("x" = 3:4, "y" = 3:4)
df4 <- data.frame("x" = 4:5, "y" = 4:5)

fun <- function(x) {
  input <- get(paste0("df", x), envir = .GlobalEnv) 
  return(input)
}

array <- 1:4
MPIresults <- mpi.remote.exec(cmd = fun, array)

But now we see the slave can not find the dataframes. One way around this is to explicitly pass them to the slave and use a method to pick the ones each slave will work on. So for four slaves, you could do:

library(Rmpi)
mpi.spawn.Rslaves(nslaves=4)

df1 <- data.frame("x" = 1:2, "y" = 1:2)
df2 <- data.frame("x" = 2:3, "y" = 2:3)
df3 <- data.frame("x" = 3:4, "y" = 3:4)
df4 <- data.frame("x" = 4:5, "y" = 4:5)

fun <- function(x,df_list) {
  slave_num <- mpi.comm.rank() #give slave number
  input <- df_list[[slave_num]] 
  return(input)
}

array <- 1:4
df_list <- list(df1,df2,df3,df4)

mpi.bcast.Robj2slave(fun)
MPIresults <- mpi.remote.exec(cmd = fun, array, df_list)

Which will return your desired output

MPIresults
$slave1
  x y
1 1 1
2 2 2

$slave2
  x y
1 2 2
2 3 3

$slave3
  x y
1 3 3
2 4 4

$slave4
  x y
1 4 4
2 5 5

Upvotes: 1

Related Questions