CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sql
Andronicus
Andronicus

Reputation: 26076

StructType from Array

What I need to do?

Create schema for a DataFrame that should look like this:

root
 |-- doubleColumn: double (nullable = false)
 |-- longColumn: long (nullable = false)
 |-- col0: double (nullable = true)
 |-- col1: double (nullable = true)
...

Columns with prefix col can vary in number. Their names are stored in an array ar: Array[String].

My attempt

val schema = StructType(
    StructField("doubleColumn", DoubleType, false) ::
    StructField("longColumn", LongType, false) ::
    ar.map(item => StructField(item, DoubleType, true)) // how to reduce it?
    Nil
)

I have a problem with the commented line (4), I don't know, how to pass this array.

Upvotes: 0

Views: 509

Answers (1)

user11047646
user11047646

Reputation:

There is no need to reduce anything. You can just perpend a list of known columns: val 

val schema = StructType(Seq(
    StructField("doubleColumn", DoubleType, false),
    StructField("longColumn", LongType, false)
  ) ++ ar.map(item => StructField(item, DoubleType, true))
)

You might also

ar.foldLeft(StructType(Seq(
  StructField("doubleColumn", DoubleType, false), 
  StructField("longColumn", LongType, false)
)))((acc, name) => acc.add(name, DoubleType, true))

Upvotes: 2

Related Questions