CODEWITHSUNDEEP
pythonpandasdatabricksazure-databricks
Stramzik
Stramzik

Reputation: 306

How do I create a folder inside Datalake gen 1 while saving a pandas dataframe as csv?

I'm new to Databricks and basically I'm trying to save a pandas dataframe to a datalake storage.

Datalake is mounted

so when I save the file to a folder which is already created it works perfectly fine however when I try to save the csv file to a folder which is not created yet its not working and throwing an error that the folder does not exist. I was under the assumption that if I give a path which is not there it does create the folder by it self.

example - folders are created until snapshot so if I try the below code works perfectly fine

df.to_csv("/dbfs/mnt/test/snapshot/test.csv", index=False)

but when I try saving inside a folder which is not yet created it throws an error

df.to_csv("/dbfs/mnt/test/snapshot/2020/08/27/test.csv", index=False)

Is there a way to achieve this via code instead of manually creating folders.

Thank you in advance

Upvotes: 0

Views: 1306

Answers (1)

Robert Long
Robert Long

Reputation: 6822

You can create the folder beforehand using dbutils.fs.mkdirs():

dbutils.fs.mkdirs("/mnt/test/snapshot/2020/08/27")
df.to_csv("/dbfs/mnt/test/snapshot/2020/08/27/test.csv", index=False)

Upvotes: 1

Related Questions