CODEWITHSUNDEEP
pysparkdatabricksazure-databricksdelta-lake
TonyRyan
TonyRyan

Reputation: 328

Databricks - overwriteSchema

Multiple times I've had an issue while updating a delta table in Databricks where overwriting the Schema fails the first time, but is then successful the second time. The solution to my problem was to simply run it again, and I'm unable to reproduce at this time. If it happens again I'll come back and post the exact error message, but it was in essence a Schema Mismatch error. Has anyone else had a similar problem?

overwriteSchema = True
DF.write \
.format("delta") \
.mode("overwrite") \
.option("overwriteSchema", overwriteSchema) \
.partitionBy(datefield) \
.saveAsTable(deltatable)

Upvotes: 8

Views: 52933

Answers (1)

geens_k
geens_k

Reputation: 181

Key-value should be string, not Boolean. .option("overwriteSchema", "True") 

DF.write \
.format("delta") \
.mode("overwrite") \
.option("overwriteSchema", "True") \
.partitionBy(datefield) \
.saveAsTable(deltatable)

Upvotes: 17

Related Questions