CODEWITHSUNDEEP
rparallel-processinglapplymclapply
forecaster
forecaster

Reputation: 1159

how to track progress in mclapply in R in parallel package

My question is related to this question. However the question referenced above uses multicore package which was replaced by parallel. Most of the functions in the response cannot be replicated in the parallel package. Is there a way to track progress in mclapply. In looking at the mclapply documentation, there is a parameter called mc.silent, I'm not sure if this would be able to track progress, and if so how and where we can see the log file? I'm running on ubuntu linux OS. Please see below for a reproducible example for which I would like to tack progress.

require(parallel) 

wait.then.square <- function(xx){
  # Wait for one second
  Sys.sleep(2);
  # Square the argument 
  xx^2 } 

output <- mclapply( 1:10, wait.then.square, mc.cores=4,mc.silent=FALSE)

Any help would be greatly appreciated.

Upvotes: 6

Views: 3272

Answers (2)

thie1e
thie1e

Reputation: 3688

Thanks to the package pbmcapply you can now easily track progress of mclapply and mcmapply jobs. Just replace mclapply by pbmclapply:

wait.then.square <- function(xx) {
    Sys.sleep(2)
    xx^2 
} 

library(pbmcapply)
output <- pbmclapply(1:10, wait.then.square, mc.cores = 4)

...which will display a pretty progress bar.

The author has a nice blog post on the technical details and performance benchmarks here.

Upvotes: 11

fotNelton
fotNelton

Reputation: 3894

This is an update of my related answer.

library(parallel)

finalResult <- local({
  f <- fifo(tempfile(), open="w+b", blocking=T)
  if (inherits(parallel:::mcfork(), "masterProcess")) {
    # Child
    progress <- 0.0
    while (progress < 1 && !isIncomplete(f)) {
      msg <- readBin(f, "double")
      progress <- progress + as.numeric(msg)
      cat(sprintf("Progress: %.2f%%\n", progress * 100))
    } 
    parallel:::mcexit()
  }
  numJobs <- 100
  result <- mclapply(1:numJobs, function(...) {
    # Do something fancy here... For this example, just sleep
    Sys.sleep(0.05)
    # Send progress update
    writeBin(1/numJobs, f)
    # Some arbitrary result
    sample(1000, 1)
  })
  close(f)
  result
})

cat("Done\n")

Upvotes: 4

Related Questions