igraph shortest_paths causes copying when launched in parallel with mclapply/parallel

I need to calculate many shortest paths between a set of starting points (origin_nodes) and a set of end points (target_nodes) in a very large graph. The graph can be over 50gb in ram. I could, in theory, easily paralilize the operation by doing:

mclapply(origin_nodes, function(on) {
  shortest_paths(my_graph, on, target_nodes)
}, mc.cores = 100)

But this can crash my server if my_graph is too large because for some reason R decides it needs to copy it to each thread. The effect is that I can rarely run more than 20 or so threads in parallel, even though I have access to 200. The same happens with all other parallelization options I have tried with R. My understanding is that R should only copy if there is any sort of mutation of the data going on, but I don't see why shortest_paths should cause the graph to be mutated.

Is there any way around this issue?

Upvotes: 1

Views: 42

Answers (0)

Related Questions