CODEWITHSUNDEEP
scalamavenapache-sparkapache-spark-mllib
Dev2017
Dev2017

Reputation: 938

How to compile and build spark examples into jar?

So I am editing MovieLensALS.scala and I want to just recompile the examples jar with my modified MovieLensALS.scala.

I used build/mvn -pl :spark-examples_2.10 compile followed by build/mvn -pl :spark-examples_2.10 package which finish normally. I have SPARK_PREPEND_CLASSES=1 set.

But when I re-run MovieLensALS using bin/spark-submit --class org.apache.spark.examples.mllib.MovieLensALS examples/target/scala-2.10/spark-examples-1.4.0-hadoop2.4.0.jar --rank 5 --numIterations 20 --lambda 1.0 --kryo data/mllib/sample_movielens_data.txt I get java.lang.StackOverflowError even though all I added to MovieLensALS.scala is a println saying that this is the modified file, with no other modifications whatsoever.

My scala version is 2.11.8 and spark version is 1.4.0 and I am following the discussion on this thread to do what I am doing.

Help will be appreciated.

Upvotes: 1

Views: 2114

Answers (1)

Dev2017
Dev2017

Reputation: 938

So I ended up figuring it out myself. I compiled using mvn compile -rf :spark-examples_2.10 followed by mvn package -rf :spark-examples_2.10 to generate the .jar file. Note that the jar file produced here is spark-examples-1.4.0-hadoop2.2.0.jar.

On the other hand, the stackoverflow error was because of a long lineage. For that I could either use checkpoints of reduce numiterations, I did the later. I followed this for it.

Upvotes: 1

Related Questions