Hortonworks Oozie Spark Action

Question

Has anyone managed to successfully run a Spark action through Oozie in HDP?

I am running on HDP 2.3.2 with Oozie 4.2.0. The spark action is set to run in yarn-cluster mode. The spark job is a dummy one, with no input file.

I have first hit the error over here and managed to bypass it by removing the following from the hdfs oozie spark lib folder (in my case: /user/oozie/share/lib/lib_20151116201309/spark/):

• tachyon-0.5.0.jar
• tachyon-client-0.5.0.jar (both because of this error)
• spark-core_2.10-1.1.0.jar
• spark-graphx_2.10-1.1.0.jar
• spark-streaming_2.10-1.1.0.jar

and added the following in the workflow lib folder:

• datanucleus-api-jdo-3.2.6.jar
• datanucleus-core-3.2.10.jar
• datanucleus-rdbms-3.2.9.jar
• spark-1.4.1.2.3.2.0-2950-yarn-shuffle.jar
• spark-assembly-1.4.1.2.3.2.0-2950-hadoop2.7.1.2.3.2.0-2950.jar
• spark-examples-1.4.1.2.3.2.0-2950-hadoop2.7.1.2.3.2.0-2950.jar

Now I land on the following error:

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, Call From hdp4/10.10.1.84 to 0.0.0.0:8032 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused java.net.ConnectException: Call From hdp4/10.10.1.84 to 0.0.0.0:8032 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

I mention that:

• both in job.properties file and in the yarn config file, the job tracker/resource manager port is set to 8032
• the application runned for ~20min (I ran it multiple times and observed this timing)

Answer 1

Here are the steps that I performed to get SparkAction to work on HDP 2.3.4.

1. Backup the Oozie sharelib spark folder and delete all the jars inside except oozie-sharelib-<spark-version>.jar. In my case I simply renamed the spark folder to spark-backup and created a new spark folder, then copied the jar I mentioned.
2. Copy all of the jars from $SPARK_HOME/lib to the Oozie sharelib spark folder.
3. Restart Oozie
4. Changed yarn.resourcemanager.address to <myhost>:8032 via Ambari
5. Restarted YARN

I had more or less the same problems before, so I posted this question on the Hortonworks Community, which is what helped me to finally get it working.