CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sql
Oleg Pavliv
Oleg Pavliv

Reputation: 21162

spark write: CSV data source does not support null data type

I have an error in my code. The code is dumping some data into Redshift database.

After some investigation I found an easy way to reproduce it in spark console.

This is working fine:

scala> Seq("France", "Germany").toDF.agg(avg(lit(null))).write.csv("1.csv")
scala>

But if I replace avg with max I got an error "CSV data source does not support null data type."

scala> Seq("France", "Germany").toDF.agg(max(lit(null))).write.csv("2.csv")
java.lang.UnsupportedOperationException: CSV data source does not support null data type.

What's wrong with max ?

Upvotes: 3

Views: 9061

Answers (1)

Chandan Ray
Chandan Ray

Reputation: 2081

The error is correct as AVG returns the DOUBLE datatype

Seq("France", "Germany").toDF.agg(avg(lit(null)).alias("col1")).printSchema

enter image description here

where as MAX returns the type as null

Seq("France", "Germany").toDF.agg(max(lit(null)).alias("col1")).printSchema

enter image description here

so while you are writing the dataframe having MAX its throwing the error, if you want to save the dataframe with the max explicitely convert it into another type

Seq("France", "Germany").toDF.agg(max(lit(null)).alias("col1").cast(DoubleType)).write.csv("path")

Upvotes: 4

Related Questions