CODEWITHSUNDEEP
scalaapache-spark
Polymerase
Polymerase

Reputation: 6781

Launch spark job on-demand from code

What is the recommended way to launch a Spark job on-demand from within an enterprise application (in Java or Scala)? There is a processing step which currently takes several minutes to complete. I would like to use a Spark cluster to reduce the processing down to, let's say less than 15 seconds:

  1. Rewrite the time consuming process in Spark and Scala.
  2. The parameters would be passed to the JAR as command line arguments. The Spark job then acquires source data from a database. Do the processing and save the output in a location readable by the enterprise application.

Question 1: How to launch the Spark job on-demand from within the enterprise application? The Spark cluster (standalone) is on the same LAN but separate from the servers on which the enterprise app is running.

Question 2: What is the recommended way to transmit the processing results back to the caller code?

Question 3: How to notify the caller code about job completion (or failure such as Spark cluster down, job time out, exception in spark code)

Upvotes: 2

Views: 650

Answers (1)

Knight71
Knight71

Reputation: 2949

You could try spark-jobserver . Upload your spark.jar to the server. And from your application, you can call the job in your spark.jar using the rest interface . To know whether your job is completed or not , you can keep polling the rest interface. And when your job completes and if the result is very small you could get it from the rest interface itself. But if the result is huge it is better to save to some db.

Upvotes: 1

Related Questions