CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sql
Kishorekumar Yakkala
Kishorekumar Yakkala

Reputation: 341

Write empty DF with header to csv

Spark creates an empty file without headers when you try to create csv file using emptyDF even though header option is true(header=true)

import ss.implicits._
val df = List((1, "kishore", 22000)).toDF("id", "name", "salary")
val emptyDF = df.where("id != 1")
emptyDF.show()

emptyDF.write.option("header", true).csv("folder/filename.csv")

is it possible to create csv file with header for emptyDF?

Upvotes: 0

Views: 472

Answers (1)

Kishorekumar Yakkala
Kishorekumar Yakkala

Reputation: 341

if(emptyDF.take(1).isEmpty){
  //To create Headers on empty DF
  ss.createDataFrame(List(Row.fromSeq(emptyDF.schema.fieldNames)).asJava, StructType(emptyDF.schema.fieldNames.map{n => StructField(n, StringType)}))
    .write.option("header", false).csv("folder/filename.csv")
} else {
  emptyDF.write.option("header", true).csv("folder/filename.csv")
}

Upvotes: 1

Related Questions