CODEWITHSUNDEEP
apache-sparkmemory-managementcpu-usage
srla
srla

Reputation: 81

How can I monitor memory and CPU usage by spark application?

After running my spark application, I want to monitor its memory and cpu usage to evaluate its performance but couldn't find any option. Is it possible to monitor it? How can I monitor memory and CPU usage by spark application?

Upvotes: 1

Views: 2790

Answers (1)

moon
moon

Reputation: 1832

There are a few options:

  1. Ganglia is one
  2. If you're running on your own cluster, HDP or Cloudera both have real time CPU & memory consumption charts.
  3. If you want specific JVM metrics, then I'd recommend FlameGraph, though it's not real time.
  4. There's also Grafana, it's extremely powerful, you can track many metrics with it, and it's real time.

Upvotes: 2

Related Questions