CODEWITHSUNDEEP
azureazure-data-factoryazure-data-lake-gen2
Kumar AK
Kumar AK

Reputation: 1037

ADF Delta Copy Activity fails, delete partly copied file from data lake

I have a hourly delta load pipeline in that i have a copy activity to copy data from sql server to datalake parquet format with folder structure (YY/MM/DD/table_nameHH) in For each activity:

after copy activity i have success/failure procedure to update records in control table

enter image description here

i need to ensure if any copy activity failed in between i need to remove that partly copied file from data lake. how to put that condition to pickand delete that dynamic file in my pipeline.

Thanks in Advance

p.s i am pretty new to this tool and learning daily.

Upvotes: 0

Views: 660

Answers (1)

Leon Yue
Leon Yue

Reputation: 16401

Please ref these steps bellow:

You could add an If condition behind the Copy_DimCustomer_AC active: enter image description here

In If condition expressions, build the expression to judge if Copy_DimCustomer_AC.executionDetails.status equals "Succeeded", if ture, then the copy active succeeded:

@equals(activity('Copy_Dimcustomer_AC').output.executionDetails.status,'Succeeded')

enter image description here

True actives:

enter image description here

False active: add a delete active to delete the file and run the Log_failure_status_AC:

enter image description here

HTH.

Upvotes: 1

Related Questions