CODEWITHSUNDEEP
apache-sparkspark-graphx
Dylan Lawrence
Dylan Lawrence

Reputation: 1533

Spark GraphX get shortest path between all vertices

In Spark GraphX the ShortestPaths does not take an RDD as an input but Seq

This means the following does not work:

val paths = ShortestPaths.run(graph,graph.vertices)

However my graph is too large to call collect on and I do not want to slow this down by forcing it onto the driver only. Is there a work around for this using a map or aggregateMessages? Or am I going to have to rewrite the path finding code myself?

Upvotes: 1

Views: 1250

Answers (1)

Vidya
Vidya

Reputation: 30310

I have to confess that your goal doesn't make sense to me. Are you looking for a map of, to use plain language, Vertex -> List of (Other Vertex, Shortest path to Other Vertex)? For me, shortest path questions always involve a relatively small subset of vertices, so the Seq makes sense to me.

Regardless, I think you will need to use the Pregel object to calculate the shortest paths yourself using the Bulk Synchronous Parallel approach pioneered by the original Pregel paper from Google and Giraph from Facebook. Here is a post on doing shortest-path using BSP with Pregel in GraphX.

Upvotes: 0

Related Questions