CODEWITHSUNDEEP
rforeachparallel-processinglogfilesink
horaceT
horaceT

Reputation: 661

How to make R foreach threads write to same log file

I have a number of lengthy jobs that I want to parallelize with foreach-dopar so that each thread works on a job independent of others. I want to track the status of each thread (some may fail while others succeed) by writing to a log file using sink. The following apparently doesn't work; the log file has only one entry. 

library(foreach)
library(doParallel)
library(doSNOW)

cl = makeCluster(2, type="SOCK")
registerDoSNOW(cl)
dl = file("runlog.Rout", open="wt")
sink(dl, type="output",  append=TRUE)
sink(dl, type="message", append=TRUE)
dump <- foreach(i=1:5, 
            .errorhandling = "stop",
            .verbose=TRUE) %dopar% 
{
    beg.time = Sys.time()
    cat(as.character(beg.time), " I am running....\n", file="mylog.txt")
    # do something here.....
    end.time = Sys.time()
    del.tm = difftime(end.time, beg.time, units="mins")  
    cat("....saving output to file......\n\n", file="mylog.txt")
    save(del.tm, file = paste("I:/Rhome/H", i, ".RData", sep=""))
    return(i)
}
stopCluster(cl)
sink(type="output")
sink(type="message")

The log file has just one line :

....saving output to file......

What went wrong ?

Upvotes: 2

Views: 2221

Answers (2)

rrs
rrs

Reputation: 9913

You can also call makeCluster with the argument outfile. From the documentatino, outfile is

Where to direct the stdout and stderr connection output from the workers. "" indicates no redirection (which may only be useful for workers on the local machine). Defaults to ‘/dev/null’ (‘nul:’ on Windows). The other possibility is a file path on the worker's host. Files will be opened in append mode, as all workers log to the same file.

Upvotes: 2

Steve Weston
Steve Weston

Reputation: 19677

Although I don't really trust having multiple processes write to the same file, you may have success by using the append=TRUE option:

cat("...\n", file="mylog.txt", append=TRUE)

Without setting this option, cat will overwrite the previous contents of "mylog.txt" each time it's called.

For other approaches, see my answer here.

Upvotes: 4

Related Questions