CODEWITHSUNDEEP
apache-sparkapache-spark-sqlapache-spark-dataset
Goutham ssc
Goutham ssc

Reputation: 84

Spark driver as a REST API

Can we have one spark driver which acts as a Rest API? Using this Rest API (1 driver) i can spin up multiple executors on fly(real time). I mean when ever a new request comes to spark Driver(RestAPI) it need to spin up new executors not another new driver.

Example scenario:

Assume you have a model with 3 steps
1) Read from one set of tables and applies join and many transformations.
2) Read from second set of tables and applies join and many transformations.
3) Finally compare above dataframes and update back some records. 

Here we have 3 input values to the model.
Likewise we have 1000 combinations of input values for the model  to run.

Upvotes: 0

Views: 699

Answers (1)

ShemTov
ShemTov

Reputation: 707

Offcourse, you can use the driver as rest api.

When ever you get a request just prepare your RDD/DF then perform an action and it`ll work.

You can do it in the driver (which mean SparkContext always up and it take resources), or you can wrap it with REST Api that submit jobs to your cluster by request.(And then for each job a new SparkContext will be created).

Upvotes: 1

Related Questions