CODEWITHSUNDEEP
rparallel-processingparallel-foreach
Utsav Bhargava
Utsav Bhargava

Reputation: 121

Error in parallel process using doSNOW

Error in { : task 1 failed - "invalid connection"

Why do I get this error, every time when I try to use all 4 cores for a parallel process.

Here is the example code: 

NumberOfCluster <- 4
cl <- makeCluster(NumberOfCluster)
registerDoSNOW(cl)

fl<- file(file.choose(),"r") # file.choose() is going to locate a file(.tsv)                 
                             # of size 8 gb (RAM is 4 GB)

foreach(i=1:3) %dopar% {

  View(name_fil <- read.delim(fl,nrows = 1000000,header = TRUE))

}

Upvotes: 1

Views: 880

Answers (1)

Steve Weston
Steve Weston

Reputation: 19677

You're getting an error because file objects can't be exported to the workers. Instead, you could export the name of the file and open that file on each of the workers:

fname <- file.choose()
foreach(i=1:3) %dopar% {
  fl <- file(fname, "r")
  View(name_fil <- read.delim(fl,nrows = 1000000,header = TRUE))
}

You may run into problems using the View function next, but this should solve the "invalid connection" error.

Upvotes: 4

Related Questions