CODEWITHSUNDEEP
apache-sparkooziehortonworks-data-platformoozie-coordinatorhortonworks-sandbox
nilesh1212
nilesh1212

Reputation: 1655

Needed spark-assembly-1.5.2.-hadoop2.7.jar for Spark -Oozie workflow on HDP 2.3.2

I am trying to schedule spark 1.5.2 job on oozie 4.2.0 (HDP 2.3.x). Spark 1.5.2 has been installed externally i am not using default spark version provided by hortonworks. I am referring below post to set this up.

https://community.hortonworks.com/questions/7014/oozie-sparkaction-throwing-javalangnosuchmethoderr.html

I am struggling to find below jars.

-spark-assembly-1.5.2.2.3.4.0-3485-hadoop2.7.1.2.3.4.0-3485.jar

-spark-examples-1.5.2.2.3.4.0-3485-hadoop2.7.1.2.3.4.0-3485.jar

If you can help me with some pointers to find/download above jars it will be a great help to get started.

Upvotes: 1

Views: 1882

Answers (1)

Nirmal Ram
Nirmal Ram

Reputation: 1210

Have you checked in spark lib path

/usr/hdp/current/spark-client/lib

You can find 

[ram@IP lib]$ ls
datanucleus-api-jdo-3.2.6.jar  datanucleus-rdbms-3.2.9.jar               spark-assembly-1.6.1.2.4.2.0-258-hadoop2.7.1.2.4.2.0-258.jar  spark-hdp-assembly.jar
datanucleus-core-3.2.10.jar    spark-1.6.1.2.4.2.0-258-yarn-shuffle.jar  spark-examples-1.6.1.2.4.2.0-258-hadoop2.7.1.2.4.2.0-258.jar
[ram@IP lib]$

Then copy the needed libs to your workflows lib

eg: hadoop fs -put /usr/hdp/current/spark-client/lib/* YOUR_WORKFLOW/lib

Upvotes: 0

Related Questions