CODEWITHSUNDEEP
rdatabricksazure-blob-storage
MP12
MP12

Reputation: 35

Write a csv file from R Notebook in Databricks to Azure blob storage?

I have a R Notebook created in databricks with some code running in it. The R script creates a csv file that I need to upload to a specific storage account blob container. To access the blob storage I have a SAS (Shared Access Signature) String. I found out an R package 'AzureStor' that connects R to Azure Storage. Is there a specific command I can use to connect to blob storage using this SAS string and write the csv file directly to the container folder? I am new to databricks and trying to automate an upload process to Azure blob storage.

Upvotes: 1

Views: 2054

Answers (1)

Peter Pan
Peter Pan

Reputation: 24148

As I known, there are two ways to write a csv file from R Notebook in Databricks to Azure Blob Storage, as below.

  1. Please refer to my answer for the other SO thread How do I upload a R dataframe as a CSV file on Azure blob storage? to use AzureStor installed by install.packages("data.table") to write a R dataframe as a csv file into Azure Blob. The sample code is like as below.

    library(AzureStor)

df <- data.frame(Column1 = c('Value 1', 'Value 2', 'Value 3'),
                 Column2 = c('Value 1', 'Value 2', 'Value 3'))

account_endpoint <- "https://<your account name>.blob.core.windows.net"
account_key <- "<your account key>"
container_name <- "<your container name>"
bl_endp_key <- storage_endpoint(account_endpoint, key=account_key)
cont <- storage_container(bl_endp_key, container_name)
w_con <- textConnection("foo", "w")
write.csv(df, w_con)
r_con <- textConnection(textConnectionValue(w_con))
close(w_con)
upload_blob(cont, src=r_con, dest="df.csv")
close(con)

  2. First, please follow the offical document Data > Data Sources > Azure Blob Storage to mount a container of Azure Blob Storage to DBFS, then you can try to use the function fwrite of data.table to write data to the directory of Azure Blob container mounted. Note: you can try to run Python script by R package reticulate, please refer to my answer for the SO thread Reading csv files from microsoft Azure using R to know how to use it.

Upvotes: 4

Related Questions