CODEWITHSUNDEEP
azureazure-devopsazure-data-factoryazure-databricks
Brendan
Brendan

Reputation: 57

How to use parameter for Databricks existing cluster id in Data Factory?

For some background, I am working in a development Azure Data Factory where I use the generated ARM templates to deploy to other test/prod environments.

The problem I am having is when trying to reference an existing cluster id in my Azure Databricks linked service. This cluster id gets passed into the different accounts where the cluster does not exist. This linked service in used in multiple pipelines so I want to be able to change it in one place.

I want to be able to have a parameter which I can override during the Azure DevOps release pipeline to map to the correct environment cluster. But as Data Factory generates the ARM template I don't have much control over it.

This is an example of what the arm template looks like.

"name": "[concat(parameters('factoryName'), '/my-linked-service')]",
"type": "Microsoft.DataFactory/factories/linkedServices",
"apiVersion": "2018-06-01",
"properties": {
    "description": "Databricks connection",
    "parameters": {
        "test": {
            "type": "string"
        }
    },
    "annotations": [],
    "type": "AzureDatabricks",
    "typeProperties": {
        "domain": "https://australiaeast.azuredatabricks.net",
        "accessToken": {
            "type": "AzureKeyVaultSecret",
            "store": {
                "referenceName": "keyName",
                "type": "LinkedServiceReference"
            },
            "secretName": "secretName"
        },
        "existingClusterId": "1234-56789-abc123"
    }

Databricks Linked Service

Upvotes: 2

Views: 3825

Answers (1)

simon_dmorias
simon_dmorias

Reputation: 2473

Only certain fields are parameterised by default. But you can set a template to customise them. It's not very pretty - the full guide is here: https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment#use-custom-parameters-with-the-resource-manager-template

For your example I have a template that looks like this:

{
    "Microsoft.DataFactory/factories/linkedServices": {
        "*": {
            "properties": {
                "typeProperties": {
                    "existingClusterId": "="
                }
            }
        }
    }
}

The file must be named arm-template-parameters-definition.json and placed in the root of your repo.

Upvotes: 2

Related Questions