CODEWITHSUNDEEP
azureapache-sparkpysparkdatabricks
cincin21
cincin21

Reputation: 610

Write DataFrame from Azure Databricks notebook to Azure DataLake Gen2 Tables

I've created a DataFrame which I would like to write / export next to my Azure DataLake Gen2 in Tables (need to create new Table for this).

In the future I will also need to update this Azure DL Gen2 Table with new DataFrames.

In Azure Databricks I've created a connection Azure Databricks -> Azure DataLake to see my my files:

enter image description here

Appreciate help how to write it in spark / pyspark.

Thank you!

Upvotes: 5

Views: 10197

Answers (1)

Chandra Mouli Gupta
Chandra Mouli Gupta

Reputation: 71

I would suggest instead of writing data in parquet format, go for Delta format which internally uses Parquet format but provide other features like ACID transaction.The syntax would be

df.write.format("delta").save(path)

Upvotes: 1

Related Questions