Azure Data Factory - copy resources between subscriptions

Question

I have 25 pipelines and numerous datasets defined in Azure data factory. What is the best way to copy/replicate (not move) these pipelines/datasets/linkes services from one subscription to another? Appreciate your help!

Answer 1

The best way would be to use a Visual Studio project.

The copy wizard in the portal is handy and the 'Author and Deploy' blade is nice. But best practice would be to develop and deploy your ADF solutions in Visual Studio first. Plus source control them of course! You can then build and publish from Visual Studio to any Azure subscriptions you have access to. The publish wizard is really nice and gives you the option to choose different configuration files for different environments. Eg. Live, Test, Dev.

I've written a blog post on using ADF config files if your interested in that approach. Here: https://www.purplefrogsystems.com/paul/2017/01/using-azure-data-factory-configuration-files/

Next best way. If you don't have a VS project yet, it's possible to reverse engineer one from an existing Azure Data Factory using the Cloud Explorer panel. See below.

But be warned it won't bring down authentication values and keys for your linked services. You'll need to recreate them. Just the same as you see the JSON attribute values masked in the portal. Eg "authorization": "**********",

Side note; You'll need to use Visual Studio 2015. Currently ADF projects aren't supported in Visual Studio 2017.

The worst way would be to copy and paste all of the JSON via notepad. You need to do it in order. Linked Services > Datasets > Pipelines > Activities.

Answer 2

I suggest to use Export and Import ARM from your "Azure Data Factory"

Go to your ADF then click on "Author"

Then on top use Export "ARM Template"

You will have a zip file in witch you will have all pipeline and parameters and Linked services

How to import an existing data factory ARM template?

Answer 3

What you can do is create a git repo and sync all your pipeline, dataset and linked service over the git and use this git to import it to some other ADF.

https://azure.microsoft.com/en-us/blog/azure-data-factory-visual-tools-now-supports-github-integration/

The Other way is you can directly export the ARM template of the ADF and you can easily import in any other ADF.

Answer 4

If you need to copy it and promote the code continuously I would suggest setting up a Git Repo: https://azure.microsoft.com/en-us/blog/azure-data-factory-visual-tools-now-supports-github-integration/ We use this to promote our code into different environments. Note that if you have an existing pipeline and the name is the same, the new pipeline will over ride the old when using this method.

Answer 5

If you are using Azure Data Factory V2. There is one risk free & easy option which is ARM Template. In ADF V2, You can see Export & Import ARM Template options in the top. -> Click on Export in the ADF where you have already developed Pipelines & Activities. -> You will get one Zip File - Update the configs of the ARM Template Parametes file. -> Move to New ADF and import these two config files.

This Process, inturn replicates current version of ADF with customized configurations.

Answer 6

Azure portal gives you option to move your entire data factory to another subscription. 1) Go to azure portal and navigate to your data factory. 2) On top-left, click Move and choose subscription where you want to move it.

If your data factory is huge and you want only selected ones to be moved. Run below commands for unwanted pipelines once above step is completed.

Remove-AzureRmDataFactoryPipeline -DataFactoryName "" -Name "" -ResourceGroup ""

You can automate this using Powershell script as well very easily using AzureRm Cmdlets.