CODEWITHSUNDEEP
hadoop
Nishant Srivastava
Nishant Srivastava

Reputation: 449

Difference between task counter and job counter

Can anyone please help me understand what is the difference between task counter and job counter in map reduce?

Hadoop,The Definitive guide says that task counters are those which are updated as the task progresses and and job counter are those which are updated as the job progresses.

Is this the only difference or they have any other difference too?

Upvotes: 0

Views: 822

Answers (1)

dpsdce
dpsdce

Reputation: 5450

Task Counters

Task counters gather information about tasks over the course of their execution, and the results are aggregated over all the tasks in a job. Task counters are sent in full every time, rather than sending the counts since the last transmission, since this guards against errors due to lost messages. Furthermore, during a job run, counters may go down if a task fails for example you dont want to add up the bad_records in a split of a failed tasks. So As the task progesses and completes successfully the overall count of the task statistics is sent over to task tracker which is passed over to job tracker.

Job counters

Job counters are maintained by the jobtracker (or application master in YARN), so they don’t need to be sent across the network, unlike all other counters, They measure job-level statistics, not values that change while a task is running For example, TOTAL_LAUNCHED_MAPS counts the number of total map task launched which is just a statistics about the overall job

Upvotes: 1

Related Questions