CODEWITHSUNDEEP
hadoopmapreduce
user1192878
user1192878

Reputation: 734

How to interpret MapReduce Performance Counters

To be more specific:

  1. In task counters, the CPU spent is from proc/stat's utime + stime, so it means things like IOWait will not be counted. Is that right?
  2. Elapsed time for the whole task are a lot longer than CPU time spent counter, does it mean the node is very busy and the container not getting CPU or waiting for IO for very long time?
  3. How can I tell if a task is CPU bound or IO count just from counter?

Upvotes: 8

Views: 471

Answers (1)

vijay kumar
vijay kumar

Reputation: 2049

'CPU_MILLISECONDS' counter can give you info about - Total time spent by all tasks on CPU.

'REDUCE_SHUFFLE_BYTES' higher the number , higher the n/w utilization. (lot more opts availble like this) enter image description here

There are 4 categories of counters in Hadoop: file system, job, framework, and custom.

You can use the built-in counters to validate that:

1.The correct number of bytes was read and written
2.The correct number of tasks was launched and successfully ran
3.The amount of CPU and memory consumed is appropriate for your job and cluster nodes
4.The correct number of records was read and written

more info avalible @ https://www.mapr.com/blog/managing-monitoring-and-testing-mapreduce-jobs-how-work-counters#.VZy9IF_vPZ4 (**credits- mapr.com)

Upvotes: 1

Related Questions