Viewing a graph in Spark with GraphX and Zeppelin

Question

I'm currently working on a project using someone else's code. I understand the basic concept of how this code works, but not all of it. To that end, I'm trying to trace a small example through a run. I know I can do this using println but I would really like to visualize the graph after each operation.

Can the graph generated in spark be viewed? I'm using Zeppelin because I read it does help with this and other debugging but I was hoping to actually make a visual representation of the graph.

Is this possible?

Answer 1

Spark GraphX has no built in visualisation method, and Zeppelin provides no built in way to visualise it either. It makes sense that this may have not been a priority if you consider that GraphX (and Spark in general) has as its main purpose to work with huge datasets distributed over a cluster of machines. To visualize the full graph on your machine, you would have to collect the full graph to that single machine, and if you can easily do that, do you really need a distributed system?

If you decide the answer is still yes, there are a few examples of visualising GraphX graphs out there:

• D3.js in Databrick's proprietary notebook - should be able to do something similar in Zeppelin/Jupyter
• Graphlab - can display graphs in IPython/Jupyter notebooks
• Gephi - commonly used standalone visualization, but no direct way to get your GraphX data in there