CODEWITHSUNDEEP
amazon-web-servicesapache-sparkpysparketlaws-glue
user6107173
user6107173

Reputation: 609

Understanding AWS Glue detailed job metrics

Please see the attached screen shot of the CPU Load: Driver and Executors. It looks fine in the first 6 minutes, multiple executors are active. But after 6 minutes the chart only shows the Executor Average and Driver lines. When I put the mouse on the line, there are no usage data for all 17 executors. Does that mean all the executors are inactive after 6 minutes? How the Executor Average is calculated?

Thank you.

enter image description here

Upvotes: 1

Views: 1697

Answers (1)

user6107173
user6107173

Reputation: 609

After talked to AWS support, I finally got the answer for why after 04:07 there are no lines for individual executors but only the Executor Average and the Driver.

I was told there are 62 executors for each job, however, at each moment at most 17 executors are used. So the Executor Average is the average of different sets of 17 executors at different moment. The default CPU Load chart only shows Executor 1 to 17, not 18 to 62. In order to show other executors, you need to manually add the metrics.

Upvotes: 1

Related Questions