CODEWITHSUNDEEP
mapreduceclouderaooziecloudera-cdhoozie-workflow
hba
hba

Reputation: 7800

Why would Oozie fail a job with Error Code LimitExceededException when yarn reports that oozie launcher & mapreduce job have completed successfully?

There are a few questions similar to this on SO. However nothing has worked for me. So I am posting this question.

I am Using CDH 6.2.1

I have a workflow that has map-reduce action. The map-reduce job creates a lot of counters (I think m/r job produces ~300 counters).

I have set the cdh/yarn/config mapreduce.job.counters.max property to 8192.

I have also set the:

If I run the map-reduce job as a stand-alone yarn job (using yarn jar command on the command-line), the job completes successfully.

When I run the job as part of the workflow:

The configuration for the mapreduce job & oozie launcher as reported by yarn has the setting:

<property>
     <name>mapreduce.job.counters.max</name>
     <value>8192</value>
     <final>true</final>
     <source>yarn-site.xml</source>
</property>

Oozie web interface System-Info/OS-Env reports that the following HADOOP_CONF_DIR: /var/run/cloudera-scm-agent/process/459-oozie-OOZIE_SERVER/yarn-conf/

In that folder I can see that the mapred-site.xml also has:

    <!--'mapreduce.job.counters.max', originally set to '8192' (final), is overridden below by a safety valve-->
  <property>
    <name>mapreduce.job.counters.max</name>
    <value>8192</value>
    <final>true</final>
  </property>

However I cannot find that property in the yarn-site.xml.

I am not sure what else I can do at this point...

Upvotes: 0

Views: 208

Answers (1)

hba
hba

Reputation: 7800

This is an oozie issue which has been resolved. However, it is not available in the current version of cloudera.

I am posting this here, in case anyone else has the same issue.

Upvotes: 0

Related Questions