CODEWITHSUNDEEP
apache-spark
srinivas amara
srinivas amara

Reputation: 155

Can we submit a spark job from spark itself (may be from another spark program)?

Can anyone clarify the question asked in one of my interviews to me? It could be that the question itself is wrong, I am not sure. However, I have searched everywhere and could not find anything related to this question. The question is:

Can we run a spark job from another spark program?

Upvotes: 0

Views: 1395

Answers (2)

Jacek Laskowski
Jacek Laskowski

Reputation: 74669

Can we run a spark job from another spark program?

I'd focus on another part of the question since the following holds for any Spark program:

Can we run a Spark job from any Spark program?

That means that either there was a follow-up question or some introduction to the one you were asked.

If I were you and heard the question, I'd say "Yes, indeed!"

A Spark application is in other words a launcher of Spark jobs and the only reason to have a Spark application is to run Spark jobs sequentially or in parallel.

Any Spark application does this and nothing more.

A Spark application is a Scala application (when Scala is the programming language) and as such it is possible to run a Spark program from another Spark program (where it makes sense in general sense I put aside as there could be conflicts with multiple SparkContexts per one single JVM).

Given the other Spark application is a separate executable jar, you could launch it using Process API in Scala (as any other application):

scala.sys.process This package handles the execution of external processes.

Upvotes: 1

Sandeep Purohit
Sandeep Purohit

Reputation: 3692

Yup, you are right its not make any sense. Like we can run our application by our driver program but its same as like we are run it from any application using spark launcher https://github.com/phalodi/Spark-launcher . Except that we can't run application inside rdd closures because they run on worker nodes so it will not work.

Upvotes: 1

Related Questions