CODEWITHSUNDEEP
apache-sparkpyspark
Renato Bibiano
Renato Bibiano

Reputation: 401

dataframe.coalesce(10).write writing 1 file in S3

We are running the following code to write a table to S3:

dataframe.coalesce(10).write.mode("overwrite").parquet(destination_path)

When I check S3, it has only 1 parquet file. How can I write it to 10 files?

Upvotes: 0

Views: 1598

Answers (1)

mck
mck

Reputation: 42422

Use repartition if you want to increase number of partitions. Coalesce only decreases the number.

dataframe.repartition(10).write.mode("overwrite").parquet(destination_path)

Upvotes: 2

Related Questions