CODEWITHSUNDEEP
rloopsasynchronousparallel-processingr-future
Brigadeiro
Brigadeiro

Reputation: 2945

Asynchronous programming in R

Overview

I am writing a program (in R) that makes API calls at certain designated times. The API calls take a while, but I need the timer (main loop) to continue counting while the API call is made. To do so, I need to "outsource" the API call to another CPU thread. I believe this is possible and have looked into the future and promises packages, but haven't found a solution yet.

Reproducible Example

Let's run a for loop that counts from 0 to 100. When the counter (i) gets to 50, it has to complete a resource-intensive process (calling the function sampler, which samples 1 million normal distributions 10,000 times for the sake of taking up computation space). The desire is for the counter to continue counting while sampler() is doing its work on another thread.

#Something to take up computation space
sampler <- function(){
  for(s in 1:10000) sample(1000000)
}

#Get this counter to continue while sampler() runs on another thread
for(i in 1:100){
  message(i)
  if(i == 50){
    sampler()
  }
}

What I have tried (unsuccessfully)

library(future)

sampler <- function(){
  for(s in 1:10000) sample(1000000)
}

for(i in 1:100){
  message(i)
  if(i == 50){
    mySamples <- future({ sampler() }) %plan% multiprocess
  }
}

Upvotes: 3

Views: 932

Answers (1)

AEF
AEF

Reputation: 5650

It seems to me your call is only blocking while the workers are created, but not for the duration of the actual work. E.g. if do the plan() first, the counter will not block:

library(future)

sampler <- function(){
  for(s in 1:10000) sample(1000000)
}

plan(multiprocess)

for(i in 1:100){
  message(i)
  if(i == 50){
    mySamples <- future({ sampler() })
  }
}

Also note, that the runtime of sampler() is much longer than the duration of the blocking call in your code and that, after executing your code, mySamples still has the status resolved: FALSE and CPU usage is still high.

Upvotes: 3

Related Questions