CODEWITHSUNDEEP
azureazure-devopscloudazure-data-factory
IT_Guy
IT_Guy

Reputation: 89

Automated Deployment of ADF Pipelines using Azure DevOps CI/CD Pipelines

I have automated the Azure ADF Pipeline Deployment process using Azure DevOps CI/CD pipelines with the help of https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment (i.e) Deploying pipelines from DEV to PROD environment ADF. I am using ARM Templates of the ADF to deploy pipelines from one environment to another. Hence I will be having a separate ARM_Parameter.json corresponding to each environment(Dev/Prod). The Problem is each ADF pipeline may have few base parameres along with it, which is not parameterized and hence it will not be available in parameter.json. Can you guys help me to replace the Dev Values with the PROD Values in Base Parameter section under each ADF Pipelines in an automated way during this automated ADF pipeline deployment process using CI/CD Pipelines?

Upvotes: 1

Views: 1725

Answers (5)

jravenger4
jravenger4

Reputation: 163

You can use PowerShell script to update Dev values to Prod values in ARMTemplateForFactory.json by adding Task in the pipeline.

Upvotes: 0

Kamil Nowinski
Kamil Nowinski

Reputation: 545

There is another approach to publish ADF, from master (collaboration) branch. You can define (replace) value for every single node (property) in JSON file (ADF object). It will resolve your problem as you can provide a separate CSV config file per each environment (stage).

Example of CSV config file (config-stage-UAT.csv):

type,name,path,value
pipeline,PL_CopyMovies,activities[0].outputs[0].parameters.BlobContainer,UAT

Then just run such cmdlet in PowerShell:

Publish-AdfV2FromJson -RootFolder "$RootFolder" -ResourceGroupName "$ResourceGroupName" -DataFactoryName "$DataFactoryName" -Location "$Location" -Stage "stage-UAT"

Check this out: azure.datafactory.tools (PowerShell module)

Upvotes: 0

VijayV
VijayV

Reputation: 51

You could use Custom parameter with ARM template. The custom parameter for Pipeline could look like this:

"Microsoft.DataFactory/factories/pipelines": {
    "properties": {
        "parameters": {
                "RUN_ENVIRONMENT": "=:-:string"
        }
    }
},

Upvotes: 1

Mengdi Liang
Mengdi Liang

Reputation: 19026

Replace the Dev Values with the PROD Values in Base Parameter section

Based on your screenshot, the RUN_ENVIRONMENT is the parameter of pipeline, which means while convert to ARM template, its format like:

 "resources": [
    {
      ....
      ....
      "properties": {
        "parameters": {
          "RUN_ENVIRONMENT": {
            "type": "string",
            "defaultValue": "pro"
          }
        },...      
      },...
    }
  ]

It can not be replaced by using Override template parameters in ARM deploy task. Because it will prompt The template parameters 'environment' in the parameters file are not valid; they are not present in the original template and can therefore not be provided at deployment time.

To around this error, just install one extension and add the Replace token task into the pipeline which before ARM deploy task. And this task will replace the value of content during the build runtime:

enter image description here

For how to apply this task in our pipeline, you could have a refer to my answer1 and answer2

Upvotes: 0

Simon Zeinstra
Simon Zeinstra

Reputation: 815

I see two options:

  1. If it's only for this RUN_ENVIRONMENT parameter, you could change your parameter to variable and use the system variable @pipeline().DataFactory to determine what environment you're running in.
  2. Otherwist, you can configure the Data Factory to generate ARM Parameters for your pipeline parameter default values, but you'll have to create a custom arm-template-parameters-definition.json file. Check the documentation here

Upvotes: 1

Related Questions