Keeping the source of the MERGE statement materialized has failed repeatedly

Question

While trying to run my DLT pipelines , I am repeatedly getting this issue

and not sure whats the cause here ?

Also I don't know how the merges happen with apply changes ? Like based on sequenceCol and primary key if it matches it deletes the record or updates it? If somebody can point me right blog or documentation then that will be great !

Edit - using SCD-1 in the delta live table

Answer 1

Try to see if increasing your cluster capacity helps (or adjusting cluster configuration). This worked for me when I encountered the same error after trying to merge a large dataframe to an existing large delta table (i.e. [DELTA_MERGE_MATERIALIZE_SOURCE_FAILED_REPEATEDLY] Keeping the source of the MERGE statement materialized has failed repeatedly.).