InvocationTargetException exception from org.apache.hadoop.hbase.client.HConnectionManager.createConnection

Question

My tomcat based REST API application is not able to process request due to above mentioned error. I have tried following things so far :

1. checking if all the jar files are available or not
2. Checking permissions on all files present in tomcat/webapp/ directory
3. firewall rules
4. Hbase is availabe or not

but then also getting following exception. I am using CDH 5.3.1 which contains HBase 0.98.6. Does anyone know how to resolve this issue?

2015-03-03 05:09:02 privateLog [ERROR]          java.lang.reflect.InvocationTargetException     org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:413)
                org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:306)
                com.amazon.dao.MyDAO.<clinit>(SensorDataDAO.java:78)
                sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
                sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
                sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                java.lang.reflect.Constructor.newInstance(Constructor.java:526)
                org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:126)
                org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:74)
                org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:958)
                org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:911)
                org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485)
                org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
                org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291)
                org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
                org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288)
                org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)
                org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:580)
                org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
                org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
                org.springframework.web.context.ContextLoader.createWebApplicationContext(ContextLoader.java:276)
                org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:197)
                org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:47)
                org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4779)
                org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5273)
                org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
                org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:895)
                org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:871)
                org.apache.catalina.core.StandardHost.addChild(StandardHost.java:615)
                org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:958)
                org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1599)
                java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
                java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
                java.util.concurrent.FutureTask.run(FutureTask.java:166)
                java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                java.lang.Thread.run(Thread.java:724)   
                com.amazon.dao.MyDAO  <clinit>

---

Code which tries to establish connection is as follows:

public class MyDAO {

  protected static HConnection connection;


  static {
      Configuration conf = HBaseConfiguration.create();
      conf.addResource("hbase-site.xml");
      connection = HConnectionManager.createConnection(conf);
      // connection object is still null at this point
    try {
    } catch (Exception ex) {
      ex.printStackTrace();
    }
  }
}

Answer 1

I solved my problem. I found 2 reasons for this error. In case someone faces same error then this answer might help them :)

1. Some jar files were missing from class path hence InvocationTargetException came. Since we are using Ant based build system and not Maven based, some jar files were missing. Basically we need to put almost all jar files present in /etc/hadoop, /etc/hadoop-hdfs, /etc/hbase and /etc/zookeeper directories.

Please take a note that we are using Cloudera Hadoop distribution, if you are using some different distribution then these paths may vary.

2. Second major problem was that we did not included hdfs-site.xml file while creating configuration object. Since we are using name node HA feature, this file is a must. IF this file is not present then hbase cannot connect to proper name node service.

Now connection related code looks like this :

public class MyDAO {

  protected static HConnection connection;

  static {
      Configuration conf = HBaseConfiguration.create();
      conf.addResource("hbase-site.xml");
      conf.addResource("hdfs-site.xml");
      conf.addResource("core-site.xml");
      connection = HConnectionManager.createConnection(conf);
    try {
    } catch (Exception ex) {
      ex.printStackTrace();
    }
  }
}

Answer 2

Can you try setting this into your conf?

config.set("hbase.zookeeper.quorum", "zookeeper-ip:port");

If this works then we can check if the hbase-site.xml you are setting into conf has the correct details or not.