CODEWITHSUNDEEP
javahadoopelasticsearchjarapache-spark
Phineas Dashevsky
Phineas Dashevsky

Reputation: 613

Remove Jars from Spark class path

I am using the ElasticSearch Spark Jar. But for some reason when I start up my sparkcontext, it also picks up the elastic-hive jar from

/usr/lib/hive/lib/elasticsearch-hadoop-hive-2.1.0.jar

This is causing elasticrdd issues where the two conflict.

Does anyone know how that Jar gets into Spark and how I can remove it from the spark classpath preferably before I start the context?

Thank you.

Upvotes: 2

Views: 2663

Answers (1)

Phineas Dashevsky
Phineas Dashevsky

Reputation: 613

It turns out this is the default configuration in the conf/spark-env.sh which loads that directory. This is easily solved by loading a different folder with the jars you want and not the ones you don't want. So instead of this:

SPARK_DIST_CLASSPATH="$SPARK_DIST_CLASSPATH:/usr/lib/hive/lib/*"

This would work:

SPARK_DIST_CLASSPATH="$SPARK_DIST_CLASSPATH:/usr/lib/hive/lib/spark/*"

assuming you copied the relevant jars into that folder

Upvotes: 2

Related Questions