CODEWITHSUNDEEP
scalaapache-sparksbtaerospikesbt-assembly
Gur
Gur

Reputation: 39

Use local .jar as a dependency for spark application written in scala

I am building a spark application to ingest data from Aerospike key value store using aerospike-spark client. I have followed the instructions here https://github.com/aerospike/aerospark to build a .jar file.
Now, I am plannig to use this .jar in another spark application by providing it in spark-submit as:

bin/spark-submit --jars my_jar.jar

Now, what should I include in my build.sbt file in order to accomplish this?

Upvotes: 1

Views: 553

Answers (1)

Michael Zhang
Michael Zhang

Reputation: 36

In this case, you are looking to use aerospark as a library in your own spark app. Inside the aerospark project, you can run command: sbt publish-local, which will publish the jar to ivy2 local repository.

Then treat aerospike-spark as a standard library by adding to libraryDependencies in your build.sbt file.

Upvotes: 2

Related Questions