CODEWITHSUNDEEP
azure-blob-storageazure-databricksazure-data-lake-gen2
sandy
sandy

Reputation: 293

Get the latest data from ADLS Gen 2 blob storage to table mounted in Azure DataBricks

I have created an unmanaged table in Azure DataBricks using mount path as below:

CREATE TABLE <Table-Name> using org.apache.spark.sql.parquet OPTIONS (path "/mnt/<folder>/<subfolder>/")

Source of mount path is parquet files stored in ADLS Gen2. I see if the underlying data is changed in ADLS Gen 2 blob storage path, it is not reflected in the unmanaged table created in ADB. This ADB table still holds the data which was available in blob storage at time of creation of table Is there any way to get the latest data from blob storage into table in ADB?

Upvotes: 0

Views: 545

Answers (1)

Himanshu Kumar Sinha
Himanshu Kumar Sinha

Reputation: 1806

There are many who suggest to use ,

   REFRESH TABLE <table-name>

https://docs.databricks.com/data/tables.html#update-a-table

But it never worked for me .

The below think it worked .

yourdataframe.write.mode("overwrite").saveAsTable("test_table")

Upvotes: 0

Related Questions