CODEWITHSUNDEEP
azure-keyvaultazure-databricks
czajek
czajek

Reputation: 699

KeyVault -> Databricks automatic integration

I have followed Create an Azure Key Vault-backed secret scope to integrate Databricks with Key Vault and all works ok. Unfortunately this requires manual intervention, which breaks our 'full automated infrastructure' approach. Is there any way to automate this step?

Upvotes: 1

Views: 911

Answers (3)

Alex Ott
Alex Ott

Reputation: 87069

Update May 2023: now it’s possible to use service principal to create a secret scope from azure keyvault: https://learn.microsoft.com/en-us/azure/databricks/release-notes/product/2023/april#create-an-azure-key-vault-backed-secret-scope-with-a-service-principal

You can use Databricks Terraform provider to create secret scope baked by the Azure KeyVault. But because of Azure limitations it should be done by using user’s AAD token (usually using azure cli). Here is the working snippet for creation of the secret scope from existing KeyVault:

terraform {
  required_providers {
    databricks = {
      source = "databrickslabs/databricks"
      version = "0.2.9"
    }
  }
}

provider "azurerm" {
  version = "2.33.0"
  features {}
}

data "azurerm_databricks_workspace" "example" {
  name                = var.workspace_name
  resource_group_name = var.resource_group
}

provider "databricks" {
  azure_workspace_resource_id = data.azurerm_databricks_workspace.example.id
}

data "azurerm_key_vault" "example" {
  name                = var.keyvault_name
  resource_group_name = var.resource_group
}

resource "databricks_secret_scope" "example" {
  name = data.azurerm_key_vault.example.name
  keyvault_metadata {
    resource_id = data.azurerm_key_vault.example.id
    dns_name    = data.azurerm_key_vault.example.vault_uri
  }
}

variable resource_group {
  type        = string
  description = "Resource group to deploy"
}

variable workspace_name {
  type = string
  description = "The name of DB Workspace"
}

variable keyvault_name {
  type = string
  description = "The name of DB Workspace"
}

Upvotes: 0

MartinJaffer-MSFT
MartinJaffer-MSFT

Reputation: 713

UPDATE: You create a Databricks-backed secret scope using the Databricks CLI (version 0.7.1 and above). Alternatively, you can use the Secrets API.

It does not appear that Azure Key Vault backed secret scope creation has a publicly available API call, unlike the Databricks backed secret scope creation. This is backed by the 'Note' on the secret scopes doc page:

Creating an Azure Key Vault-backed secret scope is supported only in the Azure Databricks UI. You cannot create a scope using the Secrets CLI or API.

A request for the feature you are asking for was made last year, but no ETA was given.

I took a look at the request made by the UI page. While the form data is simple enough, the headers and security measures make programmatic access impractical. If you are dead-set on automating this part, you could use one of those tools which automates the cursor around the screen and clicks things for you.

Upvotes: 1

Guilherme Ramos Gouveia
Guilherme Ramos Gouveia

Reputation: 1

Now it is possible, but you can't use a service principal token. It must be a user token which hinder automation.

Refer to Microsoft Docs: https://learn.microsoft.com/en-us/azure/databricks/security/secrets/secret-scopes#create-an-azure-key-vault-backed-secret-scope-using-the-databricks-cli

Upvotes: 0

Related Questions