CODEWITHSUNDEEP
apache-sparkcloudera-cdhapache-phoenix
user7437747
user7437747

Reputation: 11

Unable to save spark dataframe in phoenix

I am writing below code to save spark dataframe to phonenix:

phoenix_df.write.format("org.apache.phoenix.spark").mode(SaveMode.Overwrite).options(Map("table"->"TEST","zkUrl"->"defabc10d")).save()

using spark 1.6.0.

However, job is failing with below error:

java.lang.NoSuchMethodError: org.apache.hadoop.hbase.HTableDescriptor.setValue(Ljava/lang/String;Ljava/lang/String;)Lorg/apache/hadoop/hbase/HTableDescriptor

Please provide some solution is anybody faced same issue?

Upvotes: 1

Views: 401

Answers (1)

Samson Scharfrichter
Samson Scharfrichter

Reputation: 9067

Looks like your Phoenix connector was built against a specific version of the HBase client (e.g. V1.3), but the HBase client bundled in CDH is a different version (V1.2) and there are breaking API changes.

  1. Find out which HBase client is required by your Phoenix connector
  2. Download that client from the HortonWorks repo
  3. Add these custom JARs to your driver/executors CLASSPATH, just like you did with the Phoenix connector (e.g. --jars)
  4. Make sure that your custom JARs override the default Cloudera HBase client with --conf spark.yarn.user.classpath.first=true

Upvotes: 1

Related Questions