CODEWITHSUNDEEP
apache-spark
Luckylukee
Luckylukee

Reputation: 595

Getting the hivecontext from a dataframe

I am creating a hivecontext instead of sqlcontext to create adtaframe

val conf=new SparkConf().setMaster("yarn-cluster")
val context=new SparkContext(conf)
//val sqlContext=new SQLContext(context)
val hiveContext=new HiveContext(context)
val data=Seq(1,2,3,4,5,6,7,8,9,10).map(x=>(x.toLong,x+1,x+2.toDouble)).toDF("ts","value","label")
//outdta is a dataframe
data.registerTempTable("df")
//val hiveTest=hiveContext.sql("SELECT * from df where ts < percentile(BIGINT ts, 0.5)")
val ratio1=hiveContext.sql("SELECT percentile_approx(ts, array (0.5,0.7)) from df")

I need to get the exact hive context from ratio1 and not again create a hivecontext from the povidedsql context in the dataframe, I don't know why spark don't give me a hivecontext from dataframe and it just gives sqlcontext.

Upvotes: 0

Views: 560

Answers (1)

Raphael Roth
Raphael Roth

Reputation: 27373

If you use HiveCOntext, then the runtime-type of df.sqlContext is HiveContext (HiveContext is a subtype of SQLContext), therefore you can do:

val hiveContext = df.sqlContext.asInstanceOf[HiveContext]

Upvotes: 1

Related Questions