CODEWITHSUNDEEP
azureapache-spark-sqlazure-blob-storageazure-databricks
kanishk kashyap
kanishk kashyap

Reputation: 63

file transfer from DBFS to Azure Blob Storage

I need to transfer the files in the below dbfs file system path:

%fs ls /FileStore/tables/26AS_report/customer_monthly_running_report/parts/

To the below Azure Blob

dbutils.fs.ls("wasbs://"+blob.storage_account_container+"@"
  + blob.storage_account_name+".blob.core.windows.net/")

WHAT SERIES OF STEPS SHOULD I FOLLOW? Pls suggest

Upvotes: 2

Views: 2077

Answers (1)

restlessmodem
restlessmodem

Reputation: 448

The simplest way would be to load the data into a dataframe and then to write that dataframe into the target.

df = spark.read.format(format).load("dbfs://FileStore/tables/26AS_report/customer_monthly_running_report/parts/*")
df.write.format(format).save("wasbs://"+blob.storage_account_container+"@" + blob.storage_account_name+".blob.core.windows.net/")

You will have to replace "format" with the source file format and the format you want in the target folder.

Keep in mind that if you do not want to do any transformations to the data but to just move it, it will most likely be more efficient not to use pyspark but to just use the az-copy command line tool. You can also run that in Databricks with the %sh magic command if needed.

Upvotes: 1

Related Questions