CODEWITHSUNDEEP
hadoopapache-sparkhbaseapache-zookeeperhortonworks-data-platform
Sangeeta Rawat
Sangeeta Rawat

Reputation: 309

ERROR TableInputFormat: Java.lang.NullPointerException at org.Apache.Hadoop.hbase.TableName.valueOf

I am trying to read data from HBase using Spark. Versions I'm using is Spark 1.3.1 and Hbase 1.1.1. I am getting following error

ERROR TableInputFormat: java.lang.NullPointerException                                                              
    at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:417)                                                              
    at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:159)                                                              
    at org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:101)                                      
    at org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:91)                                                     
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)                                                        
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)                                                        
    at scala.Option.getOrElse(Option.scala:120)                                                                                   
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:217)                                                                         
    at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32)                                             
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)                                                        
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)                                                        
    at scala.Option.getOrElse(Option.scala:120)                                                                                   
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:217)                                                                         
    at org.apache.spark.ShuffleDependency.<init>(Dependency.scala:82)                                                             
    at org.apache.spark.rdd.ShuffledRDD.getDependencies(ShuffledRDD.scala:80)                                                     
    at org.apache.spark.rdd.RDD$$anonfun$dependencies$2.apply(RDD.scala:206)                                                      
    at org.apache.spark.rdd.RDD$$anonfun$dependencies$2.apply(RDD.scala:204)                                                      
    at scala.Option.getOrElse(Option.scala:120)                                                                                   
    at org.apache.spark.rdd.RDD.dependencies(RDD.scala:204)                                                                       
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal(DAGScheduler.scal

The code is as follows

 public static void main( String[] args )
{
    String TABLE_NAME = "Hello";
    HTable table=null;
    SparkConf sparkConf = new SparkConf();
    sparkConf.setAppName("Data Reader").setMaster("local[1]");
    sparkConf.set("spark.executor.extraClassPath", "$(hbase classpath)");

    JavaSparkContext sparkContext = new JavaSparkContext(sparkConf);

    Configuration hbConf = HBaseConfiguration.create();
    hbConf.set("zookeeper.znode.parent", "/hbase-unsecure");
    try {
         table = new HTable(hbConf, Bytes.toBytes(TABLE_NAME));

    } catch (IOException e) {

        e.printStackTrace();
    }

    JavaPairRDD<ImmutableBytesWritable, Result> hBaseRDD = sparkContext
            .newAPIHadoopRDD(
                    hbConf,
                    TableInputFormat.class,
                    org.apache.hadoop.hbase.io.ImmutableBytesWritable.class,
                    org.apache.hadoop.hbase.client.Result.class);
    hBaseRDD.coalesce(1, true);
    System.out.println("Count "+hBaseRDD.count());
    //.saveAsTextFile("hBaseRDD");
    try {
        table.close();
        sparkContext.close();
    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }
}

I am not able to resolve the Issue. I am using Hortonworks Sandbox for this.

Upvotes: 0

Views: 1722

Answers (1)

Simone Pessotto
Simone Pessotto

Reputation: 1581

You have write:

try {
     table = new HTable(hbConf, Bytes.toBytes(TABLE_NAME));

} catch (IOException e) {

     e.printStackTrace();
}

IF YOU'RE USING 1.1.1 api:

In the devapidocs I can see only two constructor:

protected HTable(ClusterConnection conn, BufferedMutatorParams params) For internal testing.

protected HTable(TableName tableName, ClusterConnection connection, TableConfiguration tableConfig, RpcRetryingCallerFactory rpcCallerFactory, RpcControllerFactory rpcControllerFactory, ExecutorService pool) Creates an object to access a HBase table.

The constructor of params for the first constructor is: BufferedMutatorParams(TableName tableName)

and TableName has no constructor.

So you have to initialize your HTable like this:

table = new HTable(hbConf, new bufferedMutatorParams(TableName.valueOf(TABLE_NAME))

IF YOU'RE USING 0.94 API:

The constructors of HTBale are:

HTable(byte[] tableName, HConnection connection) Creates an object to access a HBase table. HTable(byte[] tableName, HConnection connection, ExecutorService pool) Creates an object to access a HBase table.

HTable(org.apache.hadoop.conf.Configuration conf, byte[] tableName) Creates an object to access a HBase table.

HTable(org.apache.hadoop.conf.Configuration conf, byte[] tableName, ExecutorService pool) Creates an object to access a HBase table.

HTable(org.apache.hadoop.conf.Configuration conf, String tableName) Creates an object to access a HBase table.

So, look ath the last, you need only to pass the String name not the bytes[]

table = new HTable(hbConf, TABLE_NAME);

it should be ok.

Upvotes: 2

Related Questions