CODEWITHSUNDEEP
scalacsvapache-spark
Ben Haim Shani
Ben Haim Shani

Reputation: 265

Writing CSV file using Spark and scala - empty quotes instead of Null values

I'm using spark 2.4.1 and scala, and trying to write DF to csv file. it seems that in case of null values ,the csv contains "". Is it possible to remove those empty quotes?

 val data = Seq(
      Row(1, "a"),
      Row(5, "z"),
      Row(5, null)
    )

    val schema = StructType(
      List(
        StructField("num", IntegerType, true),
        StructField("letter", StringType, true)
      )
    )

    var df = spark.createDataFrame(
      spark.sparkContext.parallelize(data),
      schema
    )
  df.write.csv("location/")

The output seems like:

1,a
5,z
5,""

And I want it will be:

1,a
5,z
5,

What should I do?

Thanks!

Upvotes: 4

Views: 3568

Answers (2)

OldWolfs
OldWolfs

Reputation: 626

You can use options of the writer see CSV specific options(SaveMode is not related to answer);

 df.write
   .option("nullValue", null)
   .mode(SaveMode.Overwrite)
   .csv("location/")

Upvotes: 9

Lucas Vinicius Teodoro
Lucas Vinicius Teodoro

Reputation: 11

Try this:

df.write.option("nullValue",None).save("location/")

Upvotes: -1

Related Questions