CODEWITHSUNDEEP
rparallel-processing
Bear Bile Farming is Torture
Bear Bile Farming is Torture

Reputation: 5123

maximum number of parallel processes using future package of R

library(future)
plan(multiprocess)
for (i in 1:20) {
  background_process <-  future({
     Sys.sleep(1000)
  })
}
# takes way too long to get here

I have a XEON 8 core, 16 thread CPU

I expected the above code to return control to the parent process in a matter of seconds, the time I expect it would take to create the 20 processes. But it took much longer than expected. So I inspected the number of R processes that my machine was running as this code was running. As it turns out, there are only 16 R processes running in the background.

So the question is, why does the code only able to create 16 processes and thus blocking the parent process while it waits to create the other 4 processes?

EDITED:

Ah, I see. Running plan(multiprocess) only creates 16 processes. How do I get it to create more then?

Upvotes: 0

Views: 1762

Answers (1)

r2evans
r2evans

Reputation: 160437

If you read A Future for R: A Comprehensive Overview in a little detail, this is towards the middle:

"If nothing else is specified, all available cores on the machine will be utilized, cf. parallel::detectCores(). For more details, please see help("availableCores", package = "future").

So your XEON 8 core, 16 thread CPU is going to start with 16 processes.

I believe you can use something like this to get more:

plan(multisession, workers = 17)
# I've seen this too, not sure when this is necessary
# plan(tweak(multisession, workers = 17))

Upvotes: 3

Related Questions