CODEWITHSUNDEEP
databricksdelta-live-tables
marritza
marritza

Reputation: 52

Databricks DeltaLivetable Silver not refreshing

we have a data import running from sql server into delta live tables.

synapse imports data into landing zone.

databricks then into bronze and silver tables. we use cdc processing version one as described here: https://docs.databricks.com/delta-live-tables/cdc.html#requirements

all requirements are met. new and changed records go into landing zone and bronze. new files (json in landing, parquet in bronze) are created each.

for new records, an additional file with the new record is created in silver as well with the new record. when i query, result is correct.

but if an existing record changes, there is no new record created for it in silver to reflect the change.

this part should update the silver table:

dlt.apply_changes(
    target = f"silver_{task['ObjectName']['objectNameval']}", 
    source = f"bronze_{task['ObjectName']['objectNameval']}_v",
    keys = task['LakehouseSilverSettings']['primaryKeysval'], 
    sequence_by = task['LakehouseSilverSettings']['sequenceColumnval']
)

so the silver tables dont update for changed records. what am i missing to get silver tables updated?

Upvotes: 1

Views: 588

Answers (1)

marritza
marritza

Reputation: 52

i found the answer, this was missing:

except_column_list = ["operation", "sequenceNum"],
stored_as_scd_type = 1

as described here: https://docs.databricks.com/delta-live-tables/cdc.html#requirements

Upvotes: 0

Related Questions