CODEWITHSUNDEEP
azureupsert
user2137271
user2137271

Reputation: 11

ADF upsert error locating key column in the interim table

I have a pipeline with copy data task where I upsert some data from On-Prem SQL into Synapse. The setup is such that I first establish the list of tables and then, in ForEach, I dynamically set Key Columns (from a variable) and do an upsert. The strange part is that occasionally copy data task fails with this error

ErrorCode=SqlUpsertKeyColumnsInvalid,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Sql upsert key column '[some-column-name]' does not exist in the table '[some-schema-name].[InterimTable_9688bbfd-bb01-4a10-b6f6-9832bcb12512]'.,Source=Microsoft.DataTransfer.ClientLibrary,'

When I re-run the pipeline, the error goes away for one table but may occur for another table. There is about 100 tables involved in this copy operation and the amount of data being upserted is not substantial (few hundred records if not less).

Thank you in advance!

Upvotes: 1

Views: 512

Answers (1)

Kabo
Kabo

Reputation: 1

If you do it by looping with parallel executions, the variable might be overwritten by other parallel executions. To avoid this, you need to deactivate parallel executions or execute another pipeline in the loop. The secondary pipeline needs the parameters used in the loop and won't have them overwritten by parallel loops.

Upvotes: 0

Related Questions