CODEWITHSUNDEEP
rparallel-processingmpi
jackb
jackb

Reputation: 704

R - Error in Rmpi with snow

I'm trying to execute an MPI cluster over 3 different computers inside a local area network with the following R code:

library(plyr)
library(class)
library(snow)
cl <- makeCluster(spec=c("localhost","ip1","ip2"),master="ip3")

but I'm getting an error:

Error in mpi.comm.spawn(slave = mpitask, slavearg = args, nslaves = count,  : 
  Calloc could not allocate memory (18446744071562067968 of 4 bytes)
Warning messages:
1: In if (nslaves <= 0) stop("Choose a positive number of slaves.") : [...]
2: In mpi.comm.spawn(slave = mpitask, slavearg = args, nslaves = count,  :
  NA produced by coercition

What is this error due? I couldn't find any relevant topic on the current subject.

Upvotes: 2

Views: 2571

Answers (2)

Sagar Naikwadi
Sagar Naikwadi

Reputation: 1

You can even try out executing the code in cluster nodes by following:

  1. Create a file with name nodelist -> Write down the machine names inside that one below the other.

  2. Using mpirun try the following command in terminal : mpirun -np (no.of processes) -machinefile (path where your nodelist file is present) Rscript (filename.R). Ignore round braces.

  3. By default it will take the first node as the master and spawn the process to rest of the nodes including itself as slaves.

Upvotes: 0

Steve Weston
Steve Weston

Reputation: 19677

When calling makeCluster to create an MPI cluster, the spec argument should either be a number or missing, depending on whether you want the workers to be spawned or not. You can't specify the hostnames, as you would when creating a SOCK cluster. And in order to start workers on other machines with an MPI cluster, you have to execute your R script using a command such as mpirun, mpiexec, etc., depending on your MPI installation, and you specify the hosts to use via arguments to mpirun, not to makeCluster.

In your case, you might execute your script with:

$ mpirun -n 1 -H ip3,localhost,ip1,ip2 R --slave -f script.R

Since -n 1 is used, your script executes only on "ip3", not all four hosts, but MPI knows about the other three hosts, and will be able to spawn processes to them.

You would create the MPI cluster in that script with:

cl <- makeCluster(3)

This should cause a worker to be spawned on "localhost", "ip1", and "ip2", with the master process running on "ip3" (at least with Open MPI: I'm not sure about other MPI distributions). I don't believe the "master" option is used with the MPI transport: it's primarily used by the SOCK transport.

You can get lots of information about mpirun from its man page.

Upvotes: 4

Related Questions