CODEWITHSUNDEEP
apache-sparkjdbcapache-phoenix
Kshitij Kulshrestha
Kshitij Kulshrestha

Reputation: 2072

Apache Phoenix JDBC driver not found on spark executors

I'm connecting to apache phoenix via thick client and have already provided phoenix-client.jar to --files but still executor is unable to find the driver.

Upvotes: 0

Views: 258

Answers (1)

Kshitij Kulshrestha
Kshitij Kulshrestha

Reputation: 2072

So the question is how spark load driver class automatically on driver and executor nodes ?

Spark look for the java.sql.Driver file user META-INF directory in all the jars added to it’s classpath. This file contains the driver class name, spark reads and load them by itself.

In the phoenix client jar, java.sql.Driver file in META-INF was missing, so I created the file java.sql.Driver with content org.apache.phoenix.jdbc.PhoenixDriver

and placed under META-INF directory and re-created the jar.

By putting this updated jar, spark was able to load the driver by itself.

Upvotes: 1

Related Questions