CODEWITHSUNDEEP
pythonpysparkdatabricksdelta-lake
Blue Clouds
Blue Clouds

Reputation: 8161

Truncate delta table in Databricks using python

Delta table delete operation is given here for Python and SQL, and truncate using SQL is given here. But I cannot find the documentation for Python truncate table.

How to do it for delta table in Databricks?

Upvotes: 6

Views: 10613

Answers (2)

Faitus Joseph
Faitus Joseph

Reputation: 69

I faced the below issue Azure Synapse Analytics when working in Synapse notebooks.

AnalysisException: Table does not support truncates

Truncate operation is not supported in delta lake tables because when we create a delta table in Synapse, it's doesn't create an actual physical table, but it creates files(parquet) in ADLS. You can delete the table in the below 2 ways

  1. Use the DELETE query as below

    query = "DELETE FROM tablename" spark.sql(query)

  2. Delete the parquet files from the 'warehouse' folder in ADLS.

enter image description here

Upvotes: 0

Alex Ott
Alex Ott

Reputation: 87279

Not everything is exposed as a function for Python or Java/Scala. Some operations are SQL-only, like OPTIMIZE for example. If you want to truncate table, you have two choices:

  1. Use
spark.sql("TRUNCATE TABLE <name>")

or

spark.sql("TRUNCATE TABLE delta.`<path>`")
  1. Emulate truncate with read + write empty dataframe in overwrite mode:
df = spark.read.format("delta").load("<path>")
df.limit(0).write.mode("overwrite").format("delta").save("<path>")

Upvotes: 12

Related Questions