CODEWITHSUNDEEP
apache-sparkhbaseamazon-emrapache-phoenix
Christopher Ferris
Christopher Ferris

Reputation: 11

Load only part of HBase/Phoenix table as Spark Datafrom

I am using the following code in Spark to load specified columns of my HBase/Phoenix table into a Spark Dataframe. I can specify the columns I want to load, but can I specify which rows? Or do I have to load all rows? 

import org.apache.hadoop.conf.Configuration
import org.apache.spark.SparkContext
import org.apache.spark.sql.SQLContext
import org.apache.phoenix.spark._

sc.stop()

val sc = new SparkContext("local", "phoenix-test")
val df = sqlContext.phoenixTableAsDataFrame(
     "TABLENAME", Array("ROWKEY", "CF.COL1","CF.COL2","CF.COL3"), conf = configuration
     )

Upvotes: 1

Views: 85

Answers (1)

kc2001
kc2001

Reputation: 5249

You can add a predicate to your call to restrict which rows get retrieved, e.g.,

val df = sqlContext.phoenixTableAsDataFrame(
     "TABLENAME", Array("ROWKEY", "CF.COL1","CF.COL2","CF.COL3"),
     conf = configuration,
     predicate = Some("ROWKEY IN ('1', '2')")
     )

Upvotes: 0

Related Questions