CODEWITHSUNDEEP
scalaapache-sparkhbase
angelcervera
angelcervera

Reputation: 4199

hbase-spark for Spark 2

I want to do a full Scan on hbase from Spark 2 using Scala.

I don't have a fixed catalog definition so libraries as SHC are not an option.

My logical choice was to use hbase-spark, that is working fine in Spark 1.6

In addition to the poor documentation about this library in previous versions, my surprise has been when checking the last HBase releasees, for example tag 2.0, hbase-spark is gone! but still in the master.

So my questions are:

thx!

Upvotes: 3

Views: 451

Answers (2)

YFl
YFl

Reputation: 1429

@bp2010 already answered part of the question.

Regarding the HBase Spark see below. It works with spark 2.

There are some options that don't demand a fixed catalog from client code:

  1. HBase Spark Source code with examples are here: https://github.com/apache/hbase-connectors/tree/master/spark/hbase-spark Here you can see explanations about the repositories: https://github.com/apache/hbase-connectors/tree/master/spark/hbase-spark

  2. Apache Phoenix Spark connector https://phoenix.apache.org/phoenix_spark.html

I'm not sure if it helps you, since the table must be mapped to a Phoenix table. If you have Phoenix, and you problem is writing the catalog from code, but you can standardize types in HBase Table, for a full scan this can be the way to go. Otherwise, go with option 1.

Upvotes: 0

bp2010
bp2010

Reputation: 2472

Seems hbase-spark module was removed from the hbase project for v2.0 release

https://issues.apache.org/jira/browse/HBASE-18817

Upvotes: 1

Related Questions