CODEWITHSUNDEEP
apache-sparksbt
jaywalker
jaywalker

Reputation: 1146

sbt - Building Spark takes too much time

I am trying to build Apache Spark from the modified source using sbt. I actually have made just a few changes to the spark core, all the other modules remained untouched. However, each time I build spark by using the command 

sbt/sbt assembly

it compiles and resolves all of the dependencies again. The application that I run using modified Spark source is also using only the core Spark functionality, nothing fancy. Is there a way to make spark recompile only the modified, store resolved dependencies in cache or any other way for that matter would could speed up the jar building process.

Upvotes: 2

Views: 1992

Answers (1)

Holden
Holden

Reputation: 7452

For compiling Spark during development you can do a few things to speed stuff up. One of them is run sbt/sbt and then you can do your assembly command in the sbt shell. You can keep the same sbt shell unless you change the build/deps (in which case you will need to do a reload). The other is, while your developing you can use compile, or even just core/compile if you are only changing core, to compile Spark (of course you still need the assembly jar to do things like use the shell). Another option would be using maven with continuous compilation as documented at http://spark.apache.org/docs/latest/building-spark.html . Hope that helps and best of luck :)

Upvotes: 2

Related Questions