CODEWITHSUNDEEP
apache-spark
FlashBang
FlashBang

Reputation: 53

How to convert dataframe to a text file in spark?

I unloaded snowflake table and created a data frame. this table has data of various datatype. I tried to save it as a text file but got an error:

Text data source does not support Decimal(10,0).

So to resolve the error, I casted my select query and converted all columns to string datatype.

Then I got the below error:

Text data source supports only single column, and you have 5 columns.

my requirement is to create a text file as follows.

"column1value column2value column3value and so on"

Upvotes: 0

Views: 2213

Answers (2)

blackbishop
blackbishop

Reputation: 32700

You need to have one column if you want to write using spark.write.text. You can use csv instead as suggested in @mck's answer or you can concatenate all columns into one before you write:

df.select(
  concat_ws(" ", df.columns.map(c => col(c).cast("string")): _*).as("value")
).write
  .text("output")

Upvotes: 1

mck
mck

Reputation: 42422

You can use a CSV output with a space delimiter:

import pyspark.sql.functions as F

df.select([F.col(c).cast('string') for c in df.columns]).write.csv('output', sep=' ')

If you want only 1 output file, you can add .coalesce(1) before .write.

Upvotes: 1

Related Questions