CODEWITHSUNDEEP
shellcronapache-spark
3xCh1_23
3xCh1_23

Reputation: 1499

Using Apache Spark Shell with Cron - how to?

I would like to use Apache Spark Shell with daily Cron. The scripts that I need to run with a shell are written in Scala. Furthermore, I would need this to be done as fast as possible (that is, without loading the shell and exiting each time, since I have a lot of scripts to run). How can this be done?

Upvotes: 2

Views: 353

Answers (1)

maasg
maasg

Reputation: 37435

spark-shell is an interactive environment, not meant to run headless jobs as asked. Although technically it should be possible to do so.

Probably a better option would be to create an application "fat JAR" file out of the code that's running on the REPL and run that using cron calling either spark-submit or plain java -cp ... my.pkg.Main

Upvotes: 3

Related Questions