CODEWITHSUNDEEP
apache-sparkapache-spark-sql
other15
other15

Reputation: 849

Spark UDF error - Schema for type Any is not supported

I’m trying to create a udf that will replace negative values in a column with 0.

My dataframe is – called df, and contains one column called avg_x. This is my code for creating a udf 

val noNegative = udf {(avg_acc_x: Double) => if(avg_acc_x < 0) 0 else "avg_acc_x"}

I get this error

java.lang.UnsupportedOperationException: Schema for type Any is not supported

df.printSchema returns 

|-- avg_acc_x: double (nullable = false)

so I don’t understand why this error is occurring?

Upvotes: 9

Views: 27532

Answers (2)

mythr
mythr

Reputation: 102

This error basically occurs when the Analyzer cannot resolve the type of udf properly. That is, the types are mixed (Int and String).

Therefore,

if(avg_acc_x < 0) 0 else avg_acc_x (Integer Type)

or

if(avg_acc_x < 0) "0" else "avg_acc_x" (String Type)

should resolve this exception.

Upvotes: 3

David Griffin
David Griffin

Reputation: 13927

It's because of the else returning a String: "avg_acc_x". Take away the quotes:

val noNegative = udf {(avg_acc_x: Double) => if(avg_acc_x < 0) 0 else avg_acc_x}

Upvotes: 19

Related Questions