CODEWITHSUNDEEP
scalahbaseapache-sparkcloudera-cdh
kpeng
kpeng

Reputation: 81

Issue Connection to HBase with Spark Shell in CDH5.1

I currently have a fresh image of CDH 5.1 for virtual box and I am running into an issue when trying to connect to HBase using spark shell. Here is the scala code:

import org.apache.hadoop.hbase.HBaseConfiguration
import org.apache.hadoop.hbase.client.{HBaseAdmin,HTable,Put,Get}
import org.apache.hadoop.hbase.util.Bytes
val conf = new HBaseConfiguration()
val admin = new HBaseAdmin(conf)

Here is the error:

java.io.IOException: java.lang.reflect.InvocationTargetException
    at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:416)
    at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:393)
    at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:274)
    at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:192)
.
.
.
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:414)
    ... 43 more
Caused by: java.lang.NoClassDefFoundError: org/cloudera/htrace/Trace
    at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:195)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:479)
    at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
    at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:83)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.retrieveClusterId(HConnectionManager.java:857)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:662)
    ... 48 more
Caused by: java.lang.ClassNotFoundException: org.cloudera.htrace.Trace
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    ... 54 more

Upvotes: 0

Views: 1470

Answers (2)

kpeng
kpeng

Reputation: 81

This issue was that I didn't have my spark configuration set up properly. I had to add the following to the spark configs:
In spark-defaults.conf:
spark.executor.extraClassPath /usr/lib/hive/lib/hive-hbase-handler.jar:/usr/lib/hbase/hbase-server.jar:/usr/lib/hbase/hbase-protocol.jar:/usr/lib/hbase/hbase-hadoop2-compat.jar:/usr/lib/hbase/hbase-client.jar:/usr/lib/hbase/hbase-common.jar:/usr/lib/hbase/lib/htrace-core.jar:/etc/hbase/conf

In spark-env.sh:
export SPARK_CLASSPATH=/usr/lib/hbase/hbase-server.jar:/usr/lib/hbase/hbase-protocol.jar:/usr/lib/hbase/hbase-hadoop2-compat.jar:/usr/lib/hbase/hbase-client.jar:/usr/lib/hbase/hbase-common.jar:/usr/lib/hbase/lib/htrace-core.jar:/etc/hbase/conf

Upvotes: 1

samthebest
samthebest

Reputation: 31515

Probably a version problem as its caused by something of the form NoSuchXXXError. Is your HBase 5.1 too?

Upvotes: 0

Related Questions