CODEWITHSUNDEEP
scalaapache-spark
Gabe
Gabe

Reputation: 236

Scala spark-shell: schema function structType type mismatch

Learning scala to work with spark and having difficulty with return types:

Code:

def createSchema (name: String) : StructType = {
  if (name == "test01") {
    StructType(
      List(
        StructField("id", StringType, true),
        StructField("score", DoubleType, true)
  ))}
}

Produces:

 error: type mismatch;
 found   : Unit
 required: org.apache.spark.sql.types.StructType

Without the argument and if condition the function works as expected.

Understand the if condition is the last evaluated expression and is setting the return type to Unit.

Have tried val definition (and other variations) without success.

Code:

def createSchema (name: String) : StructType = {
  val struct: StructType = if (name == "test01") {
    StructType(
      List(
        StructField("id", StringType, true),
        StructField("score", DoubleType, true)
  ))}
  struct
}

Produces:

error: type mismatch; 
found   : Unit
required: org.apache.spark.sql.types.StructType
var struct: StructType = if (name == "test01") {
                         ^

Appreciate any help understanding the type mismatch errors and solutions.

Solution for testing function using if (as learning exercise).

Code:

def createSchema (name: String) : StructType = {
  val struct = if (name == "test01") {
    StructType(
      List(
        StructField("id", StringType, true),
        StructField("score", DoubleType, true)
    ))
  }
  else {
    StructType(
      List(
        StructField("col1", StringType, true),
        StructField("col2", StringType, true)
    ))
  }
  struct
}

Thanks for your help and explanation.

Upvotes: 0

Views: 409

Answers (1)

Michael Heil
Michael Heil

Reputation: 18525

Because you have not defined an else condition, it returns the smallest common super type for all branches, which is Unit. Keep in mind that the if condition could also not be true which means the body of your method is empty (= Unit).

You could verify this by typing in

val struct = if (name == "test01") {
  StructType(
    List(
      StructField("id", StringType, true),
      StructField("score", DoubleType, true)
    )
  )
}

into your Scala REPL to see that is retuns:

struct: Any = ()

Declare an else condition or, as you have done it, remove the if condition.

Upvotes: 1

Related Questions