CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sqlapache-spark-1.6
Zied Hermi
Zied Hermi

Reputation: 229

dataframe too many arguments in the rdd object

I tryed to use this question to convert rdd object to dataframe in spark. The class in my use case contains more than 100 arguments (columns) 

case class MyClass(val1: String, ..., val104: String )

    val df = rdd.map({ 
      case Row(val1: String, ..., val104: String) => MyClass(val1, ..., val104)
    }).toDF("col1_name", ..., "col104_name")

I got this Error: too many arguments for unapply pattern, maximum = 22

May someone help me with a concrete example; i'm using spark 1.6 with scala . Thank you

Upvotes: 1

Views: 1578

Answers (1)

Cyrille Corpet
Cyrille Corpet

Reputation: 5305

Your problem is the limitation on case classes to 22 fields.

You need to define your case class as a more structured data type, so that there are only 22 top fields (but some of those may be case classes again).

Once you've done this, you can use your Row (which itself cannot be pattern matched with 104 fields, you will have to use row(0), row(1), ..., row(103)) to build your MyClass.

Upvotes: 2

Related Questions