CODEWITHSUNDEEP
apache-sparkhadoop-yarnoozie
hawarden_
hawarden_

Reputation: 2170

Oozie: why error log shows in mapreduce job, not in Spark job?

I submitted a oozie workflow that is a shell action, it calls spark-submit to run a Spring boot application which is a jar file. It runs on yarn in client mode.

However, I found that the all Spring log is inside oozie mapreduce job in yarn, not in Spark job itself. I don't understand why?

Upvotes: 0

Views: 169

Answers (1)

ankush1377
ankush1377

Reputation: 208

The oozie shell action is nothing but a map only job. By default, you spark job prints all log to console (from where it is being run). Given that that spark job is being submitted from within the oozie action, the logs are collated & visible within the shell action logs.

Upvotes: 1

Related Questions