CODEWITHSUNDEEP
apache-sparkpysparkdistributed-computing
Nathaniel Bubis
Nathaniel Bubis

Reputation: 2295

Monitor Spark actual work time vs. communication time

On a Spark cluster, if the jobs are very small, I assume that the clustering will be inefficient since most of the time will be spent on communication between nodes, rather than utilizing the processors on the nodes.

Is there a way to monitor how much time out of a job submitted with spark-submit is wasted on communication, and how much on actual computation?

I could then monitor this ratio to check how efficient my file aggregation scheme or processing algorithm is in terms of distribution efficiency.

I looked through the Spark docs, and couldn't find anything relevant, though I'm sure I'm missing something. Ideas anyone?

Upvotes: 2

Views: 236

Answers (1)

Katya Willard
Katya Willard

Reputation: 2182

You can see this information in the Spark UI, asuming you are running Spark 1.4.1 or higher (sorry but I don't know how to do this for earlier versions of Spark).

Here is a sample image: SparkUI Executor Monitoring

Here is the page that the image came from.

A brief summary: You can view a timeline of all the events happening in your Spark job within the Spark UI. From there, you can zoom in on each individual job and each individual task. Each task is divided into shceduler delay, serialization / deserialization, computation, shuffle, etc.

Now, this is obviously a very pretty UI but you might want something more robust so that you can check this info programmatically. It appears here that you can use the REST API to export the logging info in JSON.

Upvotes: 2

Related Questions