CODEWITHSUNDEEP
pythonazuredatabricks
user22428402
user22428402

Reputation: 23

trouble saving Databricks dataframe into my azure storage account

I have a databricks workspace with json files mounted from my gen 2, I am trying to convert the json file as parquet and save in into my storage account, I created a new directory in my gen 2 where I want my parquet file to be loaded. When I do the following code df.write.parquet("/mnt/youtubedataset/youtube/cleansed") it doesn't seem to be saving it into my gen 2 conatiner.

what am I missing?

Upvotes: 0

Views: 663

Answers (1)

Bhavani
Bhavani

Reputation: 5317

I tried to replicate the issue as follows:

I have mounted my ADLS container using below code:

dbutils.fs.mount(
    source="wasbs://<containerName>@<storageaccountName>.blob.core.windows.net/",
    mount_point="/mnt/<mountName>",
    extra_configs={
f"fs.azure.account.key.<storageaccountName>.blob.core.windows.net":"<Access-Key>"  
  }
)

enter image description here

I tried to write dataframe into parquet file format into ADLS account using below code:

data = [("Alice", 28), ("Bob", 22), ("Charlie", 35)]
columns = ["Name", "Age"]
df = spark.createDataFrame(data, columns)
df.write.mode("overwrite").parquet("/mnt/youtubedataset\youtube\cleansed")

enter image description here

It wrote successfully as mentioned in above image but when I saw in storage account there is no files in it as mentioned below:

enter image description here

As per this the folders in dbfs:/mnt/ that are not actually mounted volumes but just simple folders. That's why I have checked the location of my mount point using display(dbutils.fs.mounts()) it located the mount location as storage account as mentioned below:

enter image description here

I have added forward slash to the storage path instead of back slash as mentioned below:

df.write.mode("overwrite").parquet("/mnt/youtubedataset/youtube/cleansed")

enter image description here

The file wrote successfully into ADLS storage account:

enter image description here

Once check your mount point is located to storage account.

Upvotes: 0

Related Questions