CODEWITHSUNDEEP
linuxrwindowsparallel-processingpackage
David Smith
David Smith

Reputation: 2166

How to make R use all processors?

I have a quad-core laptop running Windows XP, but looking at Task Manager R only ever seems to use one processor at a time. How can I make R use all four processors and speed up my R programs?

Upvotes: 32

Views: 51505

Answers (7)

Tom Wenseleers
Tom Wenseleers

Reputation: 7989

If you do a lot of matrix operations and you are using Windows you can install revolutionanalytics.com/revolution-r-open for free, and this one comes with the intel MKL libraries which allow you to do multithreaded matrix operations. On Windows if you take the libiomp5md.dll, Rblas.dll and Rlapack.dll files from that install and overwrite the ones in whatever R version you like to use you'll have multithreaded matrix operations (typically you get a 10-20 x speedup for matrix operations). Or you can use the Atlas Rblas.dll from prs.ism.ac.jp/~nakama/SurviveGotoBLAS2/binary/windows/x64 which also work on 64 bit R and are almost as fast as the MKL ones. I found this the single easiest thing to do to drastically increase R's performance on Windows systems. Not sure why they don't come as standard in fact on R Windows installs.

On Windows, multithreading unfortunately is not well supported in R (unless you use OpenMP via Rcpp) and the available SOCKET-based parallelization on Windows systems, e.g. via package parallel, is very inefficient. On POSIX systems things are better as you can use forking there. (package multicore there is I believe the most efficient one). You could also try to use package Rdsm for multithreading within a shared memory model - I've got a version on my github that has unflagged -unix only flag and should work also on Windows (earlier Windows wasn't supported as dependency bigmemory supposedly didn't work on Windows, but now it seems it does) :

library(devtools)
devtools::install_github('tomwenseleers/Rdsm')
library(Rdsm)

Upvotes: 1

csgillespie
csgillespie

Reputation: 60452

As of version 2.15, R now comes with native support for multi-core computations. Just load the parallel package

library("parallel")

and check out the associated vignette

vignette("parallel")

Upvotes: 16

hangmanwa7id
hangmanwa7id

Reputation: 451

I have a basic system I use where I parallelize my programs on the "for" loops. This method is simple once you understand what needs to be done. It only works for local computing, but that seems to be what you're after.

You'll need these libraries installed:

library("parallel")
library("foreach")
library("doParallel")

First you need to create your computing cluster. I usually do other stuff while running parallel programs, so I like to leave one open. The "detectCores" function will return the number of cores in your computer.

cl <- makeCluster(detectCores() - 1)
registerDoParallel(cl, cores = detectCores() - 1)

Next, call your for loop with the "foreach" command, along with the %dopar% operator. I always use a "try" wrapper to make sure that any iterations where the operations fail are discarded, and don't disrupt the otherwise good data. You will need to specify the ".combine" parameter, and pass any necessary packages into the loop. Note that "i" is defined with an equals sign, not an "in" operator!

data = foreach(i = 1:length(filenames), .packages = c("ncdf","chron","stats"),
               .combine = rbind) %dopar% {
  try({
       # your operations; line 1...
       # your operations; line 2...
       # your output
     })
}

Once you're done, clean up with:

stopCluster(cl)

Upvotes: 45

Peter M
Peter M

Reputation: 844

I believe the multicore package works on XP. It gives some basic multi-process capability, especially through offering a drop-in replacement for lapply() and a simple way to evaluate an expression in a new thread (mcparallel()).

Upvotes: 6

ephpostfacto
ephpostfacto

Reputation: 303

On Windows I believe the best way to do this would probably be with foreach and snow as David Smith said.

However, Unix/Linux based systems can compute using multiple processes with the 'multicore' package. It provides a high-level function, 'mclapply', that performs a list comprehension across multiple cores. An advantage of the 'multicore' package is that each processor gets a private copy of the Global Environment that it may modify. Initially, this copy is just a pointer to the Global Environment, making the sharing of variable extremely quick if the Global Environment is treated as read-only.

Rmpi requires that the data be explicitly transferred between R processes instead of working with the 'multicore' closure approach.

-- Dan

Upvotes: 2

Dirk is no longer here
Dirk is no longer here

Reputation: 368181

The CRAN Task View on High-Performance Compting with R lists several options. XP is a restriction, but you still get something like snow to work using sockets within minutes.

Upvotes: 29

JD Long
JD Long

Reputation: 60746

I hear tell that REvolution R supports better multi-threading then the typical CRAN version of R and REvolution also supports 64 bit R in windows. I have been considering buying a copy but I found their pricing opaque. There's no price list on their web site. Very odd.

Upvotes: 7

Related Questions