ARM Template deployment does not remove old pipelines

Question

I'm deploying my data factory using ARM Templates. When I delete a pipeline, the old version is not deleted from the target environment. The same happens with renaming, my target environment then contains a pipeline with the old and AND a pipeline with the new name.

How do I delete old pipelines with ARM Template deployments?

Note: I only want to delete the orphaned pipelines inside the data factory, not wipe out the entire data factory.

Answer 1

I had the same issue and resolved it by leveraging an extension in the marketplace called Azure Data Factory. Within it are various tasks, one of which is called 'Azure Data Factory Delete Items'. I use that before executing the ARM template task from Microsoft and ultimetly I am left with a data factory pipeline that does not contain orphaned pipelines/datasets/linked_services/etc.

Answer 2

I ran into the same issue and followed Neil P's suggestion and looked at creating a post deployment script to remove the items. I was already using CI based on the arm template, so this extended my original CI.

As of this writing, Microsoft's power-shell script does work correctly for ADF V2.

Honestly, the approach right now is a little convoluted. I had to set up two branch artifacts. One looks at adf_publish and one that looks at master. adf_publish triggers the release pipeline, master handles the powershell script.

Then, I created a deployment powershell script in my repo on the master branch .

After that, I updated the release pipeline to include the powershell script after the resource was updated. Note that I had to reference the ARMTemplateForFactory.json directly to get the script to work.

Everything worked as expected and the downstream QA release has updated properly and removed obsolete pipelines, etc.

Answer 3

In order to delete pipelines inside the data factory, you need to run the script published by Microsoft here.

Note that this requires the use of the new Azure "Az" powershell modules, if you are using Azure DevOps requires the use of the preview V4 powershell tasks.