CODEWITHSUNDEEP
windowsrparallel-processing
Hong Ooi
Hong Ooi

Reputation: 57686

Starting a Linux PSOCK cluster from a Windows machine hangs R

I'm trying to setup a cluster on a Linux box using the parallel package. A wart is that the machine I'm using as the master is running Windows as opposed to CentOS.

After some hacking around with puttygen and plink (putty's version of ssh) I got a command string that manages to execute Rscript on (a) slave, without needing a password:

plink -i d:/hong/documents/gpadmin.ppk -l gpadmin 192.168.224.128 Rscript

where gpadmin.ppk is a private key file generated using puttygen, and copied to the slave.

I translated this into a makeCluster call, as follows:

cl <- makeCluster("192.168.224.128",
                  user="gpadmin",
                  rshcmd="plink -i d:/hong/documents/gpadmin.ppk",
                  master="192.168.224.1",
                  rscript="Rscript")

but when I try to run this, R (on Windows) hangs. Well, it doesn't hang as in crashing, but it doesn't do anything until I press Escape.

However, I can laboriously get the cluster running by adding manual=TRUE to the end of the call:

cl <- makeCluster("192.168.224.128",
                  user="gpadmin",
                  rshcmd="plink -i d:/hong/documents/gpadmin.ppk",
                  master="192.168.224.1",
                  rscript="Rscript",
                  manual=TRUE)

I then log into the slave using the above plink command, and, at the resulting bash prompt, running the string that R displayed. This suggests that the string is fine, but makeCluster is getting confused trying to run it by itself.

Can anyone help diagnose what's going on, and how to fix it? I'd rather not have to start the cluster by manually logging into 16+ nodes every time.

I'm running R 3.0.2 on Windows 7 on the master, and R 3.0.0 on CentOS on the slave.

Upvotes: 1

Views: 428

Answers (1)

Steve Weston
Steve Weston

Reputation: 19677

Your method of creating the cluster seems correct. Using your instructions, I was able to start a PSOCK cluster on a Linux machine from a Windows machine.

My first thought was that it was a quoting problem, but that doesn't seem to be the case since the Rscript command worked for you in manual mode. My second thought was that your environment is not correctly initialized when running non-interactively. For instance, you'd have a problem if Rscript was only in your PATH when running interactively, but again, that doesn't seem to be the case, since you were able to execute Rscript via plink. Have you checked if you have anything in ~/.Rprofile that only works interactively? You might want to temporarily remove any ~/.Rprofile on the Linux machine to see if that helps.

You should use outfile="" in case the worker issues any error or warning messages. You should run "ps" on the Linux machine while makeCluster is hanging to see if the worker has exited or is hanging. If it is running, then that suggests a networking problem that only happens when running non-interactively, strange as that seems.

Some additional comments:

  • Use Rterm.exe on the master so you see any worker output when using outfile="".
  • I recommend using "Pageant" so that you don't need to use an unencrypted private key. That's safer and avoids the need for the plink "-i" option.
  • It's a good idea to use the same version of R on the master and workers.
  • If you're desperate, you could write a wrapper script for Rscript on the Linux machine that executes Rscript via strace. That would tell you what system calls were executed when the worker either exited or hung.

Upvotes: 1

Related Questions