CODEWITHSUNDEEP
azuredatabricksazure-keyvaultazure-private-link
csaif7
csaif7

Reputation: 65

Azure Databricks integration with Azure key vault with private endpoint

Hoping to get some clarity on the Azure DataBricks (ADB) integration with Azure Key Vault (AKV).

1- When ADB is configured with AKV backed secret which component connects to the AKV? The control plane or the cluster (data plane) ? (I'm assuming it's control plane but not sure...)

2- If AKV is configured with Private link/endpoint and ADB cluster in deployed in a VNET ( i.e. vnet injected) it is using secure cluster connectivity option (i.e. No PIP) then will the AKV backed secret suppose to work in this deployment? (If the answer to # 1 is control plane then I don't see how this will work...?)

Thanks.

Upvotes: 1

Views: 2489

Answers (1)

Kartik Bhiwapurkar
Kartik Bhiwapurkar

Reputation: 5169

• When Azure Databricks is configured with Azure Key Vault backed secret, it is the control plane that connects to the Azure Key Vault as the notebook which calls the azure key vault stored secret scope is stored in the Azure databricks workspace since it is managed service data encrypted at rest with a Databricks-managed key.

• Also, when you configure Azure key vault by integrating it with azure private link, it will configure a private zone dns by integrating it with the private endpoint that will be created. And since, traffic between the azure key vault virtual network and the azure databricks cluster traverses over the Microsoft backbone network, it eliminates exposure from the public Internet.

• The Azure databricks cluster will be having a resource ID when it is configured and that can be selected when assigning a private endpoint in aure key vault while configuring with azure private link thus successfully importing the azure key vault backed secrets in the azure databricks cluster over a private network.

• Please find the below images for more clarity: -

Secure cluster connectivity

Please find the below links for more information: -

https://learn.microsoft.com/en-us/azure/databricks/security/keys/customer-managed-key-managed-services-azure

https://learn.microsoft.com/en-us/azure/databricks/security/secure-cluster-connectivity

https://learn.microsoft.com/en-us/azure/key-vault/general/private-link-service?tabs=portal

Upvotes: 0

Related Questions