CODEWITHSUNDEEP
apache-sparkpysparksparkapi
user27200545
user27200545

Reputation: 1

Apache Spark Submit Job Rest API Documentation

Can someone please point me to where I can get the latest regarding apache-spark rest api for submitting job wither if its python or java jobs? I came across different few articles and discussion forums but they all seem to be old.

From what I can see submitting a job involves many parameters and different options as an example:

  curl -X POST http://localhost:6066/v1/submissions/create --header "Content-    
   Type:application/json;charset=UTF-8" --data '{
    "action":"CreateSubmissionRequest",
     "appArgs":[
      "/Users/tilak/jobs/test_job.py"
   ],
   "appResource":"file:/Users/tilak/jobs/test_job.py",
   "clientSparkVersion":"2.3.3",
   "environmentVariables":{
      "SPARK_ENV_LOADED":"1"
     },
       "mainClass":"org.apache.spark.deploy.SparkSubmit",  
        "sparkProperties":{
        "spark.driver.supervise":"false",
        "spark.app.name":"Simple App",
        "spark.eventLog.enabled":"true",
        "spark.submit.deployMode":"cluster",
        "spark.master":"spark://localhost:6066"
      }
   }

Im unable to find any under the official apache spark site except for the monitoring API, nor I was able to find anything under apache spark official github

what I'm looking for is a comprehensive reference that list all possible parameters and possible values so I can figure out what would work best for my case.

Thanks

I have tried to submit a a job via the api as shown in the example above using version 3.5.2, however I keep getting the following error:


   java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module      
   @0x4bbf6d0e) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module     
   java.base does not export sun.nio.ch to unnamed module

Some posts were suggesting to add some Java Options but none talks how to do it through the rest api. Im not sure what is the name of the parameter, where its located in the api request and what are the values that need to be set in order to fix this issue.

Upvotes: 0

Views: 288

Answers (1)

srikanth holur
srikanth holur

Reputation: 780

Here is the official link from spark docs:

https://spark.apache.org/docs/latest/spark-standalone.html#rest-api

Upvotes: 0

Related Questions