CODEWITHSUNDEEP
apache-sparkpysparkapache-spark-sqlspark-csv
ranjith reddy
ranjith reddy

Reputation: 481

Spark save as CSV dont ignore spaces

I am trying to save Dataframe as CSV file, I want to retain the spaces. I am using Spark 2.1.1, But when I try to save it as CSV file, all the spaces are trimmed.

I tried these options, but they didn't work. 

option("ignoreLeadingWhiteSpace",false")
  .option("ignoreTrailingWhiteSpace", "false")

Expected CSV format

SiteNumber, batch ,DayBatchDate,RecordType   
    190000,    TBD,  12/12/2017,  +00000001

My current output:

SiteNumber, batch ,DayBatchDate,RecordType   
190000,TBD,12/12/2017,+00000001

Upvotes: 0

Views: 1309

Answers (1)

Alper t. Turker
Alper t. Turker

Reputation: 35229

ignoreLeadingWhiteSpace and ignoreTrailingWhiteSpace options for the writer have been introduced in Spark 2.2 ( SPARK-18579) so won't have effect in Spark 2.1.

Upvotes: 3

Related Questions