CODEWITHSUNDEEP
logginghadoopmapreduceslf4jlogback
DeejUK
DeejUK

Reputation: 13481

How does Logging in Hadoop Jobs Work?

How does logging in a Hadoop job work? Using SLF4J and Logback, what sort of configuration would I need to see all the logging output in one place? Does STDOUT for a Hadoop job get collated by the JobTracker?

Upvotes: 1

Views: 1580

Answers (1)

Joe23
Joe23

Reputation: 5782

The log directory on each datanode contains a sub-directory userlogs. This contains sub-diretories for recent map-task attempts. That is for each instance of a map task. Since the task attempt contains the job-id in its name you can find out what logs where created by a specific job.

The task attempt directories contain the files:

  • stderr
  • stdout
  • syslog

These contain the respective outputs.

You can access task logs from the JobTracker Web-GUI by navigating from a listed Job to its tasks, clicking on a task and selecting its output.

Upvotes: 2

Related Questions