Hive on Tez: Could not find or load main class 400

Question

I'm trying to configuring the Hive on Tez by following the formal document and the Tez installation guide. By overview my environment and configuration are as the following: -

1. Apache Hadoop/YARN version 3.1.2
2. Apache Hive version 3.1.2
3. Apache Tez version 0.9.2
4. openjdk version 1.8.0_242
5. OS: CentOS version 7.7.1908 and Debian version 10 (buster)

So far, I've achieved to start the YARN/Spark/Hive and Hive(mr) with bin/hive and bin/beeline properly. Then I move to the next step for configuring the Hive on Tez as the following information: -

Environment variable

export TEZ_CONF_DIR=/opt/tez/conf
export TEZ_JARS=/opt/tez/*:/opt/tez/lib/*
export HADOOP_CLASSPATH=$TEZ_CONF_DIR:$TEZ_JARS:$(hadoop classpath)

HDFS

hdfs dfs -mkdir -p /apps/tez \
&& hdfs dfs -put /opt/tez/* /apps/tez \
&& hdfs dfs -chmod g+w /apps/tez

tez-site.xml

<configuration>
    <property>
        <name>tez.lib.uris</name>
        <value>${fs.defaultFS}/apps/tez/,${fs.defaultFS}/apps/tez/lib/</value>
    </property>
    <property>
        <name>tez.use.cluster.hadoop-libs</name>
        <value>false</value>
    </property>
</configuration>

hive-site.xml

<configuration>
    <property>
        <name>hive.execution.engine</name>
        <value>tez</value>
    </property>

    <property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:postgresql://metastore-db:5432/metastore</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>org.postgresql.Driver</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>some-user</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>some-password</value>
    </property>
</configuration>

Start the Hive Server 2

${HIVE_HOME}/bin/hiveserver2 --hiveconf hive.server2.enable.doAs=false

There is an application displayed at Resource Manager UI and its status is FAILED, together with the hiveserver2 show me the error as the following:-

Failing this attempt.Diagnostics: [2020-03-10 08:28:06.410]Exception from container-launch.
Container id: container_1583471849377_0012_02_000001
Exit code: 1

[2020-03-10 08:28:06.413]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class 400


[2020-03-10 08:28:06.414]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class 400


For more detailed output, check the application tracking page: 
http://node-master:61688/cluster/app/application_1583471849377_0012 
Then click on links to logs of each attempt.
. Failing the application.

I also have a chance to visit the application log on YARN

The stderr mentions as Error: Could not find or load main class 400

The launch_container.sh mentions as

echo "Launching container"
exec /bin/bash -c "$JAVA_HOME/bin/java  \
-Xmx1228m \
-Djava.io.tmpdir=$PWD/tmp \
-server \
-Djava.net.preferIPv4Stack=true \
-Dhadoop.metrics.log.level=WARN  \

400 \ # <---- Suspected which may be related to the above error.

-Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator \
-Dlog4j.configuration=tez-container-log4j.properties \
-Dyarn.app.container.log.dir=/opt/hadoop/logs/userlogs/application_1583471849377_0012/container_1583471849377_0012_02_000001 \
-Dtez.root.logger=INFO,CLA \
-Dsun.nio.ch.bugLevel='' \
org.apache.tez.dag.app.DAGAppMaster \
--session \
1>/opt/hadoop/logs/userlogs/application_1583471849377_0012/container_1583471849377_0012_02_000001/stdout \
2>/opt/hadoop/logs/userlogs/application_1583471849377_0012/container_1583471849377_0012_02_000001/stderr "

I've no idea where the error Could not find or load main class 400 is from and the only one may be related is inside the launch_container.sh as mention above.

Could you please help to advise how to solve this trouble? Am I doing something wrong?

Edit:

The gist for displaying the long lines is

https://gist.github.com/charleech/8cd615c0236626882c7a3ea7532ad4d5

Answer 1

In my case, the 400 class came from a configuration parameter on file $HADOOP_HOME/etc/hadoop/mapred-site.xml and it indeed showed up on the launch_container.sh. I edited the following property in the file.

<property>
    <name>yarn.app.mapreduce.am.command-opts</name>
    <!-- <value>400</value> -->
    <value>-Xmx256m</value>
</property>

You can find further information of this parameter on this cloudera doc.