CODEWITHSUNDEEP
rlistloopsforeachparallel-processing
aaaaa
aaaaa

Reputation: 185

r - nested %dopar% loop between lists

I need to perform a nested %dopar% loop between two lists in R.

I have the loop working with a non-parallelised code, as follows:

main_lst = rep(list(list()), 10) # create main list where loop's results will be stored

lst_1 = rep(list(list()), 25) # create list no. 1

for (i in 1:length(lst_1)) {
  lst_1[[i]] = data.frame(x = seq(1:30), y = rnorm(30))
}

lst_2 = rep(list(list()), 10) # create list no. 2

for (i in 1:length(lst_2)) {
  lst_2[[i]] = data.frame(x = seq(16:30), z = rnorm(15))
}


#### Do the for loop (non parallelised)
for (h in 1:length(main_lst)) {
  for (i in 1:length(lst_1)) {

    main_lst[[h]][[i]] = merge(lst_1[[i]], lst_2[[h]][,c(1:2)], by = 'x')

  }
}

Any suggestion on how I can parallelise the above for loop? Shall I try lapply (or parlapply) instead?

Here what I tried but it does not work:

### Run in Parallel
library(foreach)
library(doParallel)

#setup parallel backend to use many processors
cores=detectCores()
cl = makeCluster(cores[1]-1) 
registerDoParallel(cl)


main_lst = foreach(h=1:length(main_lst)) %:% {
  foreach(i=1:length(lst_1)) %dopar% {

    main_lst[[h]][[i]] = merge(lst_1[[i]], lst_2[[h]][,c(1:2)], by = 'x')

  }
}


#stop cluster
stopCluster(cl)

Error in foreach(h = 1:main_lst) %:% { : "%:%" was passed an illegal right operand

Upvotes: 0

Views: 287

Answers (1)

Steve Lee
Steve Lee

Reputation: 776

I edited your just a few (deleted { and } , change the h iteration limit and allocate main_lst)

main_lst = foreach(h=1:10) %:% 
  foreach(i=1:length(lst_1)) %dopar% {
    merge(lst_1[[i]], lst_2[[h]][,c(1:2)], by = 'x')
   }

The result of foreach statement will be gathered by list automatically(Unless you set specific type like .combine = rbind.)

So you don't have to allocate it!!

Upvotes: 2

Related Questions