CODEWITHSUNDEEP
rforeachparallel-processing
Ekuah
Ekuah

Reputation: 11

Perfectly fine loop runs indefinitely when started with foreach in R

I have been looking for answers and working on a solution on this for the last week, but was not successful at all.

I want to set up a network of entities that depend on each other. Entities have certain thresholds: one which requires them to react to survive and one for their certain death. So if entity A dies, this may force entity B to act, but entity C may still be ok with the loss of A. The triggered reactions may in turn trigger reactions from formerly unaffected entities.
The loop is supposed to find and document the effects of every single entity on the network if it were to die.

My R code below works perfectly fine when executed on its own but runs until infinity if used in a foreach loop (I had it run for ~9 hours).

Since the network is randomly generated and different for each iteration, I need to go for as many randomly generated networks as possible. My idea that each parallel process generates an entire network and goes through the effects of every single entity’s death.

My code looks like this (shortened version):

library(foreach)  
library(doParallel)        

iterations <- … # number of iterations of the model
cl <- makeCluster((detectCores() -1), outfile = "")
registerDoParallel(cl)
parallel_loop <- foreach(iteration=(1:iterations), .packages=c("VGAM","network","igraph")) %dopar%{

# generate data set
data <- … # VGAM package is needed for this
adjacency_matrix <- …
data$reaction <- 1 # need this later for a while loop
documentation <- matrix(0, nrow = number_entities, ncol = 6) # for documentation (obviously)
documentation <- as.data.frame(documentation) 
some_network_analysis # network and igraph packages are needed for this
save(data, file=”data”)
save(adjacency_matrix, file =”adjacency_matrix”)

# define functions
check_effect_function <- function{… # determines the effect of the dead entity on the others, sets data$reaction[affected_entity] <- 1 if the effect on an entity is sufficiently strong to require a reaction
quick_reaction_test <- function{… # quick and dirty test if the triggered reactions do not cause other entities to react. == 1 if yes, ==0 if not
reaction_function <- function{… # a reaction function for affected entities
documentation_function <- function{… # filling data frame “documentation”

# the killing loop
for (dead_entity in 1:number_entities) {
    load(data) # so we have clean data and no remnants of earlier runs
    load(adjacency_matrix)
    adjacency_matrix[,dead_entity] <- 0 # the effect of the entity’s death
    while (sum(data$reaction) != 0) { # finding new equilibrium
        data$reaction <- 0
        effect_function() # sets data$reaction[entity] <- 1 if threshold is exceeded
        if (sum(data$reaction) != 0) {
            quick_test_function()
            if (quick_test ==1) 
                {documentation_function() # end of the loop 
            } else reaction_function()
        } else {documentation_function} # end of the loop
            if (reaction_needed == 0) {documentation_function()} # end of the loop
} # end of the killing loop

return(documentation)

} # end of foreach loop

docu_loop <- as.data.frame(do.call(rbind,lapply(parallel_loop,function(x){x}))) # to get me the documentation file

stopCluster(cl)

If I run the code manually (i.e. all of it but the foreach part), it finishes in under a minute. Once I include the foreach part, keeps running and running. Even if I just have 1 iteration, it does not stop. Even stranger, once I manually stop the calculations, R remains at a very high processor usage and significantly slows down the system. I can only stop this by restarting R completely.

FYI:
- The functions include for and while loops cycling through all entities. But that should work even in parallel, right?
- I have listed the used packages in case they interfere with the parallelization

What am I doing wrong in my foreach commands? This exact foreach code worked fine for some other arbitrary code I tried. Help is highly appreciated, thank you in advance!

++++ SOLUTION ++++
Should anyone ever have this problem as well, I have found the solution: In this bit

while (sum(data$reaction) != 0) { # finding new equilibrium
    data$reaction <- 0

I accidentally had

data$reaction <<- 0

which works fine for manual runs of the code, but does not work in parallelization. I should've been rubber ducking this part better.

Upvotes: 0

Views: 369

Answers (1)

Ekuah
Ekuah

Reputation: 11

Should anyone ever have this problem as well, I have found the solution: In this bit

while (sum(data$reaction) != 0) { # finding new equilibrium
    data$reaction <- 0

I accidentally had

data$reaction <<- 0

which works fine for manual runs of the code, but does not work in parallelization. I should've been rubber ducking this part better.

Upvotes: 1

Related Questions