CODEWITHSUNDEEP
apache-spark
data_person
data_person

Reputation: 4480

schedule spark job in spark-shell

I have a spark-scala script I am running in spark-shell

spark2-shell -i test.scala

It works fine. Any suggestions on how I can schedule this script to run every hour could be helpful.

Upvotes: 0

Views: 127

Answers (1)

BlueSheepToken
BlueSheepToken

Reputation: 6099

Actually, this is not spark related since there is no built in mechanism for that.

You can actually consider using a cron like this 

0 * * * * spark2-shell -i test.scala

or you might also want to use oozie, which is a bit more complex, but a lot easier to maintain if you have a lot of spark job to run

https://oozie.apache.org/docs/3.1.3-incubating/index.html

Upvotes: 2

Related Questions