CODEWITHSUNDEEP
apache-sparkpyspark
Suraj
Suraj

Reputation: 101

Saving dataframe records in a tab delimited file

How can I save records of a DataFrame into a tab delimited output file? The DataFame looks like below:

>>> csvDf.show(2,False)

1. |1  |Eldon Base for stackable storage shelf, platinum  |Muhammed
MacIntyre|3  |-213.25|38.94 |35   |Nunavut|Storage & Organization   
|0.8 | 
2. |2  |1.7 Cubic Foot Compact "Cube" Office Refrigerators|Barry
French      |293|457.81 |208.16|68.02|Nunavut|Appliances            
|0.58|

Upvotes: 8

Views: 25066

Answers (3)

dripp
dripp

Reputation: 147

In Spark 2.4.3 it is:

csvDf
.write
.option("sep", "\t")
.option("encoding", "UTF-8")
.csv(targetFilePath)

Upvotes: 2

Suraj
Suraj

Reputation: 101

this worked for me ...

csvDf.rdd.map(lambda x: '\t'.join(x)).coalesce(1).saveAsTextFile('/output/csv/6.csv')

Upvotes: 1

Alper t. Turker
Alper t. Turker

Reputation: 35219

Just pass delimiter option to the writer:

csvDf.write.option("delimiter", "\t").csv(output_path)

In Spark 1.6 use spark-csv package (check README for detailed instructions) with the same option:

csvDf.write.option("delimiter", "\t").format("com.databricks.spark.csv").save(output_path)

Upvotes: 15

Related Questions