CODEWITHSUNDEEP
pythonpyspark
tferrari
tferrari

Reputation: 53

Pyspark write in file doesn't not exist

I try to write in file in pyspark but I have an error, the file doesn't exist. I'm new in pyspark.

I have this code for write:

result.repartition(1).write.partitionBy('client', 'payload_type').json(OUTPUT_PATH, mode='append')

Is it possible to add a parameter to force to create folder/file if doesn't exist ?

Upvotes: 0

Views: 277

Answers (1)

tferrari
tferrari

Reputation: 53

I make a mistake, the error is not at the line of the post but this:

existing_data = spark.read.json(OUTPUT_PATH)

with:

OUTPUT_PATH = f"s3a://{BUCKET_DEST}/{SOURCE}/"

At the first execution, both folder doesn't exist, can I force create it if no exist on the read?

Upvotes: 1

Related Questions