CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sql
data_person
data_person

Reputation: 4500

Create ArrayType column in Spark scala

Structure of the Schema to be created:

|-- col1: boolean (nullable = true)
|-- col2: array (nullable = true)
|    |-- element: struct (containsNull = true)
|    |    |-- col2_1: boolean (nullable = true)
|    |    |-- col2_2: string (nullable = true)

Code to create schema:

val prodSchema = StructType(Array(StructField("col1", StringType), StructField("col2",ArrayType(Array(StructField("element",StructType(Array(StructField("col2_1",StringType)))))))))

Error:

found   : Array[org.apache.spark.sql.types.StructField]
required: org.apache.spark.sql.types.DataType
StructField("col2",ArrayType(Array(StructField("element",StructType(Array(StructField("col2_1",StringType)))))))

Any suggestions on how to correct this schema error.

Upvotes: 1

Views: 6603

Answers (3)

randal25
randal25

Reputation: 1330

I think you can write it like this:

val prodSchema =
  StructType(
    List(
      StructField("col1", BooleanType),
      StructField("col2", ArrayType(
        StructType(
          List(
            StructField("col2_1", BooleanType),
            StructField("col2_2",StringType)
          )
        )
      ))
    )
  )


prodSchema.printTreeString()

root
 |-- col1: boolean (nullable = true)
 |-- col2: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- col2_1: boolean (nullable = true)
 |    |    |-- col2_2: string (nullable = true)

Upvotes: 4

1pluszara
1pluszara

Reputation: 1528

Try this:

val schema = StructType(Seq(  
    StructField("col1",BooleanType,false),
    StructField("col2",ArrayType(StructType(Seq(  
                       StructField("col2_1",BooleanType,true),
                       StructField("col2_2",StringType,true)
                         )))
               )))

Upvotes: 1

Achilleus
Achilleus

Reputation: 1944

You could use the Schema DSL to create the schema:

val col2 = new StructType().add($"col2_1".boolean).add($"col2_2".string)
val schema = new StructType()
                 .add($"col1".boolean)
                 .add($"col2".array(col2))

schema.printTreeString()

root
 |-- col1: boolean (nullable = true)
 |-- col2: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- col2_1: boolean (nullable = true)
 |    |    |-- col2_2: string (nullable = true)

Hope it helps.

Upvotes: 0

Related Questions