CODEWITHSUNDEEP
scalaapache-spark
Vishal
Vishal

Reputation: 1492

Spark 2 to Spark 1.6

I am trying to convert following code to run on the spark 1.6 but, on which I am facing certain issues. while converting the sparksession to context 

 object TestData {
  def makeIntegerDf(spark: SparkSession, numbers: Seq[Int]): DataFrame =
    spark.createDataFrame(
      spark.sparkContext.makeRDD(numbers.map(Row(_))),
      StructType(List(StructField("column", IntegerType, nullable = false)))
    )

}

How Do I convert it to make it run on spark 1.6

Upvotes: 1

Views: 2403

Answers (2)

Assaf Mendelson
Assaf Mendelson

Reputation: 13001

The only main difference here is the use of spark which is a spark session as opposed to spark context.

So you would do something like this:

object TestData {
  def makeIntegerDf(sc: SparkContext, sqlContext: SQLContext, numbers: Seq[Int]): DataFrame =
    sqlContext.createDataFrame(
      sc.makeRDD(numbers.map(Row(_))),
      StructType(List(StructField("column", IntegerType, nullable = false)))
    )
}

Of course you would need to create a spark context instead of spark session in order to provide it to the function.

Upvotes: 1

Ramesh Maharjan
Ramesh Maharjan

Reputation: 41957

SparkSession is supported from spark 2.0 on-wards only. So if you want to use spark 1.6 then you would need to create SparkContext and sqlContext in driver class and pass them to the function.

so you can create 

val conf = new SparkConf().setAppName("simple")
val sparkContext = new SparkContext(conf)
val sqlContext = new SQLContext(sparkContext)

and then call the function as 

val callFunction = makeIntegerDf(sparkContext, sqlContext, numbers)

And your function should be as 

def makeIntegerDf(sparkContext: SparkContext, sqlContext: SQLContext, numbers: Seq[Int]): DataFrame =
    sqlContext.createDataFrame(
      sparkContext.makeRDD(numbers.map(Row(_))),
      StructType(List(StructField("column", IntegerType, nullable = false)))
    )

Upvotes: 3

Related Questions