Cannot create secret scope: Azure KeyVault is not available. Terraform

Question

I'm deploying a keyvault and a databricks resource with terraform. The resources are deploy via a Service Principal that will give contributor rights to every thing it deploys.

I also added to myself contributor role for the KeyVault and created a user that is assigned to admin group in Databricks.

I followed the documentation on how to create a secret scope on databricks, it worked with the UI with my account. But I wish to automate this with Terraform, so I added this in my tf file:

resource "databricks_secret_scope" "kv_db" {
  name = module.keyvault_gb.name

  keyvault_metadata {
    resource_id = module.keyvault_gb.id
    dns_name    = module.keyvault_gb.uri
  }
}

This is the result of my terraform plan

  # databricks_secret_scope.kv_db will be created
  + resource "databricks_secret_scope" "kv_db" {
      + backend_type = (known after apply)
      + id           = (known after apply)
      + name         = "xxxxxx"

      + keyvault_metadata {
          + dns_name    = "https://xxxxxx.vault.azure.net/"
          + resource_id = "/subscriptions/***/resourceGroups/*********/providers/Microsoft.KeyVault/vaults/xxxxxx"
        }
    }

And the result of my terraform apply

│ Error: cannot create secret scope: Azure KeyVault is not available
│ 
│   with databricks_secret_scope.kv_db,
│   on databricks.tf line 68, in resource "databricks_secret_scope" "kv_db":
│   68: resource "databricks_secret_scope" "kv_db" {

I don't understand why it does not work in terraform since I managed to create one with the UI. (https://learn.microsoft.com/en-us/azure/databricks/security/secrets/secret-scopes#create-an-azure-key-vault-backed-secret-scope-using-the-ui)

The only difference is the user used, the UI is my own user, and the terraform is a Service Principal that has contributor role on all resources deployed by it. The SP is also admin member on databricks.

I even tried to recreate the keyvault policy to be sure to add the good rights for the SP, but the policy already exists:

data "azurerm_client_config" "current" { # Using the deployement Service Principal
}

resource "azurerm_key_vault_access_policy" "this" {
  key_vault_id       = module.keyvault_gb.id
  tenant_id          = data.azurerm_client_config.current.tenant_id
  object_id          = data.azurerm_client_config.current.object_id
  secret_permissions = ["Delete", "Get", "List", "Set"]
}

│ Error: A resource with the ID "/subscriptions/***/resourceGroups/xxxxx/providers/Microsoft.KeyVault/vaults/xxxxxx/objectId/xxxxxx-xxxxx-xxxxx-xxxx-xxxxx" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for "azurerm_key_vault_access_policy" for more information.

Update: Here are the rights of the role of the SP used to deploy the KeyVault/Databricks and attempting to create the secret scope

Here is the role of my user:

Update 2: I added more roles to the SP, but still not working.

Update 3: Using @Vinay-B code and changing the keyvault name, this is what I get in terraform plan and apply:

Terraform will perform the following actions:

  # azurerm_resource_group.example will be created
  + resource "azurerm_resource_group" "example" {
      + id       = (known after apply)
      + location = "eastus"
      + name     = "demorgvk1"
    }

  # azurerm_key_vault.example will be created
  + resource "azurerm_key_vault" "example" {
      + access_policy                 = (known after apply)
      + id                            = (known after apply)
      + location                      = "eastus"
      + name                          = "kv-3-stackoverflow"
      + public_network_access_enabled = true
      + resource_group_name           = "demorgvk1"
      + sku_name                      = "standard"
      + soft_delete_retention_days    = 90
      + tenant_id                     = "***"
      + vault_uri                     = (known after apply)
    }

  # azurerm_key_vault_access_policy.example will be created
  + resource "azurerm_key_vault_access_policy" "example" {
      + id                 = (known after apply)
      + key_vault_id       = (known after apply)
      + object_id          = "4ace89c9-93bf-4041-b6e9-bc73c2fcb144"
      + secret_permissions = [
          + "Get",
          + "List",
          + "Set",
          + "Delete",
        ]
      + tenant_id          = "***"
    }

  # azurerm_resource_group.example will be updated in-place
  ~ resource "azurerm_resource_group" "example" {
        id       = "/subscriptions/***/resourceGroups/demorgvk1"
        name     = "demorgvk1"
      ~ tags     = {
          - "availability"    = "A1" -> null
          - "confidentiality" = "C1" -> null
          - "integrity"       = "I1" -> null
          - "spoke_type"      = "APPI" -> null
          - "traceability"    = "T1" -> null
        }
        # (1 unchanged attribute hidden)
    }


  # databricks_secret_scope.kv_db will be created
  + resource "databricks_secret_scope" "kv_db" {
      + backend_type = (known after apply)
      + id           = (known after apply)
      + name         = "vk-secret-scope"

      + keyvault_metadata {
          + dns_name    = (known after apply)
          + resource_id = (known after apply)
        }
    }

│ Error: cannot create secret scope: Azure KeyVault is not available
│ 
│   with databricks_secret_scope.kv_db,
│   on tests.tf line 25, in resource "databricks_secret_scope" "kv_db":
│   25: resource "databricks_secret_scope" "kv_db" {

Update 4: (Partial workaround)

I found an ugly work around, it's to automate inside the terraform the usage of the Databricks's API 2.0

// Create SP in Databricks
resource "databricks_service_principal" "sp" {
  application_id = <SP_APPLICATION_ID> 
  external_id    = <SP_OBJECT_ID> 
  force          = false 
}
//Get the admin group in Databricks
data "databricks_group" "group_admins" { display_name = "admins" }
//Assign databricks users to Admin databricks group
resource "databricks_group_member" "admin_assign" {
  group_id  = data.databricks_group.group_admins.id
  member_id = databricks_service_principal.sp.id
  }
resource "null_resource" "databricks_secret_scope_api" {
  provisioner "local-exec" {
    command = <<-EOT atoken=$(curl -X POST \
     -H "Content-Type: application/x-www-form-urlencoded" \ 
     https://login.microsoftonline.com/<TENANT_ID>/oauth2/v2.0/token \
     -d "client_id=<SP_APPLICATION_ID>" \
     -d "grant_type=client_credentials" \
     -d "scope=2ff814a6-3304-4ab8-85cb-cd0e6f879c1d%2F.default" \
     -d "client_secret=<SP_PASSWORD>" | jq --raw-output .access_token)
    curl -X POST \
      https://${<YOUR DATABRICKS BASE URL>}/api/2.0/secrets/scopes/create \
      --header "Content-type: application/json" \
      --header "Authorization: Bearer $atoken" \
      --data '{
        "scope": "<SCOPE NAME>",
        "scope_backend_type": "AZURE_KEYVAULT",
        "backend_azure_keyvault": {
          "resource_id": "<KEYVAUKT ID>",
          "dns_name": "<KEYVAULT URI>"
        }
      }'
  EOT
  }
  depends_on = [databricks_group_member.admin_assign]
}

This will run only ONCE. So if someone deletes the secret scope outsite the terraform, the state will never know it and thus never recreate it.

You can add a trigger to make it run all the time, it will destroy the terraform reference (NOT the secret scope) and create it back. It will fail because the secret scope already exists. You could improve this to run every time and list the secret scope names on databricks and create it only if you don't find it.

Answer 1

@BeGreen, I solved this by adding a Provider to initialize the databricks workspace.

provider "databricks_provider" {
  #Get the workspace_url from the output of the databricks module
  alias               = "databricks"
  host                = workspace_url
  azure_client_id     = client_id 
  azure_client_secret = sp_clientSecret_value
  azure_tenant_id     = tenant_id
}

resource "databricks_secret_scope" "kv" {
  provider = databricks_provider.databricks
  name     = "secret-kv-managed"

  keyvault_metadata {
    resource_id = data.azurerm_key_vault.main.id
    dns_name    = data.azurerm_key_vault.main.vault_uri
  }
}