CODEWITHSUNDEEP
mavenapache-sparkazure-hdinsightlivy
SiddharthaRT
SiddharthaRT

Reputation: 2297

How do I submit a jAR with dependencies to Livy?

I'm using Livy on HDInsight to submit jobs to a Spark cluster. I have my code written and compiled into a JAR, but it has multiple dependencies, some of which are from a custom repository.

How do I get Livy to resolve these dependencies by itself? I dont want to submit a fat jar because the dependencies are changing always and I dont like the ops effort involved.

Upvotes: 1

Views: 4672

Answers (1)

aggFTW
aggFTW

Reputation: 456

You may want to pass spark.jars.ivy as a parameter. Refer to https://spark.apache.org/docs/latest/configuration.html

You can pass anything that Livy supports in the /batches POST body: https://github.com/cloudera/livy#post-batches

Upvotes: 1

Related Questions