CODEWITHSUNDEEP
scalahadoopapache-sparkhiveapache-spark-sql
Mamun
Mamun

Reputation: 2534

feeding several variables from scala / spark-shell to hive table

I have 4 variables in scala / spark-shell.

S1 = (a string) = "age"
S2 = (another string) = "school"
D1 = (a double) = 0.50
D2 = (another double) = 0.75

I need to feed this to a hive table like so:

Factor Coeff

age 0.50

school 0.75

I was able to create table from scala/spark-shell:

val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)

//following works

hiveContext.sql("create table students_table (factor STRING, coeff FLOAT) stored as orc")

However, I am at loss as to how to insert these values to the hive table.

I have played to insert and update statements. I have also played with data frames. I have also tried dump the data into text files in hdfs (first turning them into RDDs), but the format came out in such a way that I was unable to then use it as fodder for hive table.

I am sure I am missing the whole idea on how to do this.

Any help is sincerely appreciated.

Upvotes: 2

Views: 522

Answers (1)

Pankaj Arora
Pankaj Arora

Reputation: 544

val input = sc.parallelize(Array((s1,D1), (s2, D2)))
case class StudentTable(factor : String, coeff : Double)
import sqlContext.implicits._
val df = input.map(x=>StudentTable(x._1, x._2)).toDF
df.saveAsTable("students_table", org.apache.spark.sql.SaveMode.Append)

Upvotes: 4

Related Questions