CODEWITHSUNDEEP
scalaapache-sparkemramazon-emr
user8708025
user8708025

Reputation:

Running Scala Spark Jobs on Existing EMR

I have Spark Job aggregationfinal_2.11-0.1 jar which I am running on my machine.the composition of it is as follows :

package deploy
    object FinalJob {
      def main(args: Array[String]): Unit = {
        val spark = SparkSession
          .builder()
          .appName(s"${this.getClass.getSimpleName}")
          .config("spark.sql.shuffle.partitions", "4")
          .getOrCreate()

    //continued code
    }
    }

When I am running this code in local mode, it is running fine but when I am deploying this on the EMR cluster with putting its jar in main node.It is giving error as :

ClassNotFoundException : deploy.FinalJob

What am i missing here?

Upvotes: 0

Views: 664

Answers (2)

Bhanu-Bigdata Developer
Bhanu-Bigdata Developer

Reputation: 131

try to unjar it to some folder and look for target/classes using the below command jar -xvf myapp.jar. If the target classes is not containing the class you are executing then there is an issue with the way you build your jar. I would recommend maven assembly to be in your pom for packaging.

Upvotes: 0

Przemek
Przemek

Reputation: 228

The best option is to deploy your uber jar(you can use sbt assembly plugin to build jar) to s3 and add spark step to EMR cluster. Please check:http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-submit-step.html

Upvotes: 0

Related Questions