CODEWITHSUNDEEP
jdbcapache-sparkpyspark
Frank B.
Frank B.

Reputation: 1873

spark.driver.extraClassPath Multiple Jars

I'm trying to use Spark via Python to access (via JDBC) a PostGres database and a MSSQL database in the same session. In the spark-defaults.conf file I can get one or the other to work but not both.

These two work independently:

spark.driver.extraClassPath /Users/myusername/spark-1.6.1-bin-hadoop2.4/lib/postgresql-9.4.1208.jre6.jar

spark.driver.extraClassPath /Users/myusername/spark-1.6.1-bin-hadoop2.4/lib/sqljdbc4.jar

I tried these three and neither works (I get the "no suitable driver" error): 

spark.driver.extraClassPath /Users/myusername/spark-1.6.1-bin-hadoop2.4/lib/

spark.driver.extraClassPath /Users/myusername/spark-1.6.1-bin-hadoop2.4/lib/postgresql-9.4.1208.jre6.jar sqljdbc4.jar

spark.driver.extraClassPath /Users/myusername/spark-1.6.1-bin-hadoop2.4/lib/postgresql-9.4.1208.jre6.jar /Users/myusername/spark-1.6.1-bin-hadoop2.4/lib/sqljdbc4.jar

Thanks in advance.

Upvotes: 14

Views: 22562

Answers (1)

Yuval Itzchakov
Yuval Itzchakov

Reputation: 149538

If you want to use multiple jars you need to chain them together. If you're running Linux, the chain operator is :, on Windows its ;.

For example, on Linux your extraClassPath would be:

spark.driver.extraClassPath /Users/myusername/spark-1.6.1-bin-hadoop2.4/lib/postgresql-9.4.1208.jre6.jar:/Users/myusername/spark-1.6.1-bin-hadoop2.4/lib/sqljdbc4.jar

On Windows:

spark.driver.extraClassPath /Users/myusername/spark-1.6.1-bin-hadoop2.4/lib/postgresql-9.4.1208.jre6.jar;/Users/myusername/spark-1.6.1-bin-hadoop2.4/lib/sqljdbc4.jar

Upvotes: 18

Related Questions