CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sql
santhosh sandy
santhosh sandy

Reputation: 11

spark.createDataFrame () not working with Seq RDD

CreateDataFrame takes 2 arguments , an rdd and schema.

my schema is like this

val schemas= StructType( Seq( StructField("number",IntegerType,false), StructField("notation", StringType,false) ) )

in one case i am able to create dataframe from RDD like below:

`val data1=Seq(Row(1,"one"),Row(2,"two"))

val rdd=spark.sparkContext.parallelize(data1)

val final_df= spark.createDataFrame(rdd,schemas)`

In other case like below .. i am not able to

`val data2=Seq((1,"one"),(2,"two"))

val rdd=spark.sparkContext.parallelize(data2)

val final_df= spark.createDataFrame(rdd,schemas)`

Whats wrong with data2 for not able to become a valid rdd for Dataframe?

but we can able to create dataframe using toDF() with data2 but not CreateDataFrame.

val data2_DF=Seq((1,"one"),(2,"two")).toDF("number", "notation")

Please help me understand this behaviour.

Is Row mandatory while creating dataframe?

Upvotes: 0

Views: 863

Answers (1)

Raphael Roth
Raphael Roth

Reputation: 27383

In the second case, just do :

val final_df = spark.createDataFrame(rdd)

Because your RDD is an RDD of Tuple2 (which is a Product), the schema is known at compile time, so you don't need to specify a schema

Upvotes: 1

Related Questions