Overriding ADF trigger runtime state parameter in Azure DevOps pipeline

Question

We want to put a temporary measure in place which involves keeping an ADF trigger stopped in our QA environment, but keep them started in production.

I'm trying to override this parameter in our Azure DevOps pipeline where we have defined variables for QA and production.

I've followed the same syntax of where we've overriden other parameters but get the following error when Azure DevOps is doing a deploy (it builds successfully):

##[error]Deployment template validation failed: 'The template parameters 'TriggerName_properties_runtimeState' in the parameters file are not valid; they are not present in the original template and can therefore not be provided at deployment time. The only supported parameters for this template are ... . Please see https://aka.ms/arm-pass-parameter-values for usage details.'.

This is what is in my template deploy-adf.yml:

parameters:
...
- name: triggerNameTriggerRunState 
  type: string

jobs:
- deployment: Deploy_${{ parameters.environment }}
  strategy:
    runOnce:
      deploy:
        steps:
          ...
          - task: AzureResourceManagerTemplateDeployment@3
            displayName: 'Deploying to ${{ parameters.environment }}'
            inputs:
              deploymentScope: 'Resource Group'
              azureResourceManagerConnection: ${{ parameters.serviceConnection }}
              subscriptionId: '$(subscriptionId)'
              action: 'Create Or Update Resource Group'
              resourceGroupName: '${{ parameters.adfResourceGroupName }}'
              location: '$(location)'
              templateLocation: 'Linked artifact'
              csmFile: '$(Pipeline.Workspace)/build/adf-artifact-${{ parameters.version}}/ARMTemplateForFactory.json'
              csmParametersFile: '$(Pipeline.Workspace)/build/adf-artifact-${{ parameters.version}}/ARMTemplateParametersForFactory.json'
              overrideParameters: >-
                ...
                -TriggerName_properties_runtimeState "${{ parameters.triggerNameTriggerRunState }}"
              deploymentMode: 'Incremental'

As mentioned, it builds successfully, however when I look at the generated Artifacts file for ARMTemplateForFactory.json:

    {
        "name": "[concat(parameters('factoryName'), '/TriggerName')]",
        "type": "Microsoft.DataFactory/factories/triggers",
        "apiVersion": "2018-06-01",
        "properties": {
            "annotations": [],
            "runtimeState": "Started",
            "pipelines": [
                {
                    "pipelineReference": {
                        "referenceName": "Pipeline name",
                        "type": "PipelineReference"
                    },
                    "parameters": {
                        "sourceFolder": "[parameters('TriggerName_properties_Pipeline name_parameters_sourceFolder')]",
                        "sourceFile": "[parameters('TriggerName_properties_Pipeline name_parameters_sourceFile')]"
                    }
                }
            ],
            "type": "BlobEventsTrigger",
            "typeProperties": {
                "blobPathBeginsWith": "/container/blobs/sub_directory",
                "ignoreEmptyBlobs": true,
                "scope": "[parameters('TriggerName_properties_typeProperties_scope')]",
                "events": [
                    "Microsoft.Storage.BlobCreated"
                ]
            }
        },
        "dependsOn": [
            "[concat(variables('factoryId'), '/pipelines/Pipeline name')]"
        ]
    },

And I notice that the runtimeState is not parameterised. I'm not too sure how this file gets generated. Does this mean it's not possible to override runtimeState? Or there's something I need to do in the build file that I'm missing.

Note, this is the excerpt from our Azure Devops build file:

...

variables:
  - template: ./vars/vars.qa.yml

stages:
  - stage: Build_And_Publish_ADF_Artifacts
    displayName: "Build and publish ADF"
    jobs:
      - job: Build_Adf_Arm_Template
        displayName: "Build ADF ARM template"
        steps:
          - checkout: self
            persistCredentials: "true"
            fetchDepth: 0

          - task: gitversion/setup@0
            displayName: GitVersion setup
            inputs:
              versionSpec: "5.x"

          - task: gitversion/execute@0
            displayName: Determine next version
            name: Version

          - script: |
              echo ##vso[build.updatebuildnumber]$(SEMVER)
              echo ##vso[task.setvariable variable=imageTag;isOutput=true]$(SEMVER)
            displayName: Update Build Number
            name: buildNumber

          - task: NodeTool@0
            inputs:
              versionSpec: "16.x"
            displayName: "Install Node.js"

          - task: Npm@1
            inputs:
              command: "install"
              workingDir: "$(Build.SourcesDirectory)/data_factory"
              verbose: true
            displayName: "Install NPM Package"
          - task: Npm@1
            displayName: "Validate ADF Code"
            inputs:
              command: "custom"
              workingDir: "$(Build.SourcesDirectory)/data_factory"
              customCommand: "run build validate $(Build.SourcesDirectory)/data_factory $(adfResourceId)"
          - task: Npm@1
            displayName: "Validate and Generate ARM template"
            inputs:
              command: "custom"
              workingDir: "$(Build.SourcesDirectory)/data_factory"
              customCommand: 'run build export $(Build.SourcesDirectory)/data_factory $(adfResourceId) "ArmTemplate"'
          - task: PublishPipelineArtifact@1
            displayName: Download Build Artifacts - ADF ARM templates
            inputs:
              targetPath: "$(Build.SourcesDirectory)/data_factory/ArmTemplate" #replace with the package.json folder.
              artifact: "adf-artifact-$(Build.BuildNumber)"
              publishLocation: "pipeline"

...

Answer 1

The trigger property runtimeState is not parametrized by default. In order to parametrize this property, one needs to modify the ARM template parameter configuration. Edit parameter configuration This will open the file arm-template-parameters-definition.json for editing, where the triggers-related bit of the file will initially look like the following, where the runtimeState property is not parametrized:

    "Microsoft.DataFactory/factories/triggers": {
    "properties": {
        "pipelines": [
            {
                "parameters": {
                    "*": "="
                }
            },
            "pipelineReference.referenceName"
        ],            
        "pipeline": {
            "parameters": {
                "*": "="
            }
        },
        "typeProperties": {
            "scope": "="
        }
    }
}

To parametrize the runtimeState, we just need to add "runtimeState": "=" to the above. The final specification looks like this:

    "Microsoft.DataFactory/factories/triggers": {
    "properties": {
        "pipelines": [
            {
                "parameters": {
                    "*": "="
                }
            },
            "pipelineReference.referenceName"
        ],
        "runtimeState": "=",
        "pipeline": {
            "parameters": {
                "*": "="
            }
        },
        "typeProperties": {
            "scope": "="
        }
    }
}

Now, you can specify the desired value for the runtimeState for each trigger in each environment (dev, test/qa, prod) in the environment specific ARM template parameter json file. The default version of this file can be generated for the development environment via the Export ARM template utility on the same page as the Edit parameter configuration utility. The generated file can be edited to create the parameter files for test and production environments. The name and location of the environment-specific template parameter file then need to be passed to the deployment task (and pre- and post-deployment script tasks) in the deploy-adf.yml template while deploying to the respective environments.

Answer 2

This issue happens when the parameters specified in the ARMTemplateParametersForFactory.json (ARM template parameters file) and the ARMTemplateParametersForFactory.json ARM template file have a mismatch.

TriggerName_properties_runtimeState in the parameters file are not valid. You need to check inside the parameters section of your ARM template if this parameter is specified or used.

If the parameter is defined in Parameters file and not used in the main template. It will show this error.

I can reproduce the similar issue.

To Solve this issue, you can define the parameters in the ARMTemplateParametersForFactory.json file.

For example:

Parameters file:

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
      "TriggerName_properties_runtimeState":{
           "value":"Started"
      }
    }
  }

Main template:

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "TriggerName_properties_runtimeState": {
            "type": "string",
            "defaultValue": "Started"
        },
    }
.....

If you aren't using this parameter in your template, then remove it from your parameters file ARMTemplateParametersForFactory.json and it should fix the issue.

To get more information about ARM Template parameters, you can refer to this doc: Tutorial: Use parameter files to deploy your ARM template