CODEWITHSUNDEEP
pythonapache-spark
CHETAN RAJPUT
CHETAN RAJPUT

Reputation: 151

How to submit a jar using python script in apache livy?

I have made a jar in which my main code is (written in java spark). I also have some external jars needed to run my main code ( main jar) How can I submit and execute my jar using python script in Livy ? (using spark-submit)

I just need a syntax of how to do spark-submit where my python script should export my main jar ( i am using 'request' in python for livy )

update

Getting some error in batch submit in livy .

spark logic : spark logic

livy code livy code in which above file is accessed

spark-submit (terminal) spark-submit command

livy log

livy log file

Everything working fine when i submit code using /statements but now i am taking my code through a file using /batches , my SparkContext isnt working. Also using response = request.get() while using /statements i was able to get output in my response.json() but now using /batches (considering i am externally initializing SparkContext) i am not able to get output in my json though it is showing in livy log.

Upvotes: 0

Views: 1491

Answers (1)

Neha Jirafe
Neha Jirafe

Reputation: 751

This is a working example from my recent project.The jars are on a S3 bucket. You can edit the code to use jars from a file location on your cluster all nodes Please refer to the following snippet, master_dns is the dns address of the livy master.

import json, requests
def spark_submit(master_dns):
        host = 'http://' + master_dns + ':8998'
        data = {"conf": {"spark.hadoop.fs.s3a.impl": "org.apache.hadoop.fs.s3a.S3AFileSystem"},
                'file': "s3://<your driver jar>",
                "jars": ["s3://<dependency>.jar"]
        headers = {'Content-Type': 'application/json'}
        print("Calling request........")
        response = requests.post(host + '/batches', data=json.dumps(data), headers=headers)
        print(response.json())
        return response.headers

Upvotes: 1

Related Questions