CODEWITHSUNDEEP
rparallel-processingdoparallel
user5363218
user5363218

Reputation:

Parallel computation: Loading packages in each thread only once

I am currently working with some large datasets, so parallelizing the workflows is the only way to go.

I need to load some packages to each thread once at the beginning (i.e: for(this.thread in threads) { #load some packages }.

Unfortunately , I'm not sure how to do that.

The following code further illustrates my problem, where I am trying to use the pipe operator from magrittr in a %dopar% :

. 

library(parallel)
library(doParallel)
library(foreach)
library(magrittr)


# Generate some random data and function :
# -----------------------------------------

randomData = runif(10^3)
randomFunction = function(x) {x * (2^x) } 

randomData[1] %>% randomFunction #Works



# And now ... The parallel part :
# --------------------------------

myCluster = makeCluster(6)
registerDoParallel(myCluster)


# Test that the do par is up and running: 
foreach(i = randomData) %dopar% { i }


# Use magrittr pipe operator: 
# Error in { : task 1 failed - "could not find function "%>%""
foreach(i = randomData) %dopar% { i %>% randomFunction }


# Load the library at each loop: (ie: length(data) times !)
# Other than unnecessarily loading the library (length(data) - numberOfThreads) times, 
# it works nicely
foreach(i = randomData) %dopar% { library(magrittr);  i %>% randomFunction }


# Now try without re-loading: 
# Tararaa - (ie: Works nicely)
foreach(i = randomData) %dopar% { i %>% randomFunction }

.

Any ideas?

Upvotes: 6

Views: 4237

Answers (2)

mlegge
mlegge

Reputation: 6913

The doParallel package inherits some handy low level functions from parallel including clusterCall which executes the function once on each node.

I had the exact same problem and solved it by doing:

library(doParallel)
myCluster = makeCluster(6)
registerDoParallel(myCluster)
clusterCall(myCluster, function() library(magrittr))

You can also use the argument .packages:

foreach(i = 1:5, .packages = "magrittr") %dopar% {i %>% runif}

Upvotes: 15

user56134
user56134

Reputation: 1

You could try this:

foreach(i = randomData,.packages=c("magrittr")) %dopar% {
  i %>% randomFunction
}

Upvotes: -1

Related Questions