CODEWITHSUNDEEP
rmultithreadingforeachdoparallel
Pablo Boswell
Pablo Boswell

Reputation: 845

R foreach doParallel with 1 worker/thread

Well, I don't think anyone understood the question...

I have a dynamic script. Sometimes, it will iterate through a list of 10 things, and sometimes it will only iterate through 1 thing.

I want to use foreach to run the script in parallel when the items to iterate through is greater than 1. I simply want to use 1 core per item to iterate through. So, if there are 5 things, I will parallel across 5 threads.

My question is, what happens when the list to iterate through is 1?

Is it better to NOT run in parallel and have the machine maximize throughput? Or can I have my script assign 1 worker and it will run the same as if I had not told it to run in parallel at all?

Upvotes: 0

Views: 1370

Answers (1)

JustGettinStarted
JustGettinStarted

Reputation: 834

So lets call the "the number of things you are iterating" iter which you can set dynamically for different processes

Scripting the parallelization might look something like this

if(length(iter)==1){
  Result <- #some function
} else {
  cl <- makeCluster(iter)
  registerDoParallel(cl)
  Result <- foreach(z=1:iter) %dopar% {
    # some function 
  }
  stopCluster(cl)
}

Here if iter is 1 it will not invoke parallelization otherwise it will assign cores dynamically according to the number of iter. Note that if you intend to embed this in a function, makeCluster and registerDoParallel cannot be called within a function, you have to set them outside a function.

Alternatively you register as many clusters as you have nodes, run the foreach dynamically and the unused clusters will just remain idle.

EDIT: It is better to run NOT to run in parallel if you have only one operation to iterate through. If only to avoid additional time incurred by makeCluster(), registerDoParallel() and stopCluster(). But the difference will be small compared to going parallel with one worker. Modified code above adding conditional to screen for the case of just one worker. Please provide feedback bellow if you need further assistance.

Upvotes: 1

Related Questions