Databricks API 2.0- Create Secret Scope - TEMPORARILY_UNAVAILABLE

Question

I am automating the deployment of an infrastructure containing an Azure Databricks instance. To be able to use the Azure Blob Storage from within Databricks I want to create a Secret Scope via the Databricks REST API 2.0 in my DevOps Pipeline running a Python job.

When I try to create the secret scope, I get the response

{"message":"Authentication is temporarily unavailable. Please try again later.", "error_code": "TEMPORARILY_UNAVAILABLE"}

I was already able to create a databricks access token using the API, i.e. the endpoint /token/create worked perfectly.

I am authenticating to databricks using the Code from the answer to this question: https://stackoverflow.com/a/61826488/2196531
This is how I am able to create a token and how I try to generate the scope:

import requests
import adal
import json

# set variables 
clientId = "<Service Principal Id>"
tenantId = "<Tenant Id>"
clientSecret = "<Service Principal Secret>"
subscription_id = "<Subscription Id>"
resource_group = "<Resource Group Name>"
databricks_workspace = "<Databricks Workspace Name>"
dbricks_url = "<Databricks Azure URL>"

# Acquire a token to authenticate against Azure management API
authority_url = 'https://login.microsoftonline.com/'+tenantId
context = adal.AuthenticationContext(authority_url)
token = context.acquire_token_with_client_credentials(
    resource='https://management.core.windows.net/',
    client_id=clientId,
    client_secret=clientSecret
)
azToken = token.get('accessToken')

# Acquire a token to authenticate against the Azure Databricks Resource
token = context.acquire_token_with_client_credentials(
    resource="2ff814a6-3304-4ab8-85cb-cd0e6f879c1d",
    client_id=clientId,
    client_secret=clientSecret
)
adbToken = token.get('accessToken')

# Format Request API Url
dbricks_api = "https://{}/api/2.0".format(dbricks_url)

# Request Authentication
dbricks_auth = {
    "Authorization": "Bearer {}".format(adbToken),
    "X-Databricks-Azure-SP-Management-Token": azToken,
    "X-Databricks-Azure-Workspace-Resource-Id": ("/subscriptions/{}/resourceGroups/{}/providers/Microsoft.Databricks/workspaces/{}".format(subscription_id, resource_group, databricks_workspace) )
    }

# Creating a databricks token
payload = {
    "comment": "This token is created by API call"
}
requests.post(f"{dbricks_api}/token/create", headers=dbricks_auth, json=payload)
# works

# Creating a databricks secret scope
payload = {
    "scope": "my-databricks-secret-scope",
    "initial_manage_principal": "users"
}
requests.post(f"{dbricks_api}/secrets/scopes/create", headers=dbricks_auth, json=payload)
# returns {"message":"Authentication is temporarily unavailable. Please try again later.", "error_code": "TEMPORARILY_UNAVAILABLE"}

Databricks is running in westeurope.
Python 3.8.5 x64
Packages used in the Code snippet

• adal-1.2.4
• requests-2.24.0

Is there a problem with the databricks API or am I doing something wrong?

Answer 1

According to my test, when we use the Databricks Rest API to create Secret Scope, we should use the person access token.

For example

1. Create a service principal

az login
az ad sp create-for-rbac -n "MyApp"

2. Code

import requests
import adal
import json

# set variables 
clientId = "<Service Principal Id>"
tenantId = "<Tenant Id>"
clientSecret = "<Service Principal Secret>"
subscription_id = "<Subscription Id>"
resource_group = "<Resource Group Name>"
databricks_workspace = "<Databricks Workspace Name>"
dbricks_url = "<Databricks Azure URL>"

# Acquire a token to authenticate against Azure management API
authority_url = 'https://login.microsoftonline.com/'+tenantId
context = adal.AuthenticationContext(authority_url)
token = context.acquire_token_with_client_credentials(
    resource='https://management.core.windows.net/',
    client_id=clientId,
    client_secret=clientSecret
)
azToken = token.get('accessToken')

# Acquire a token to authenticate against the Azure Databricks Resource
token = context.acquire_token_with_client_credentials(
    resource="2ff814a6-3304-4ab8-85cb-cd0e6f879c1d",
    client_id=clientId,
    client_secret=clientSecret
)
adbToken = token.get('accessToken')

# Format Request API Url
dbricks_api = "https://{}/api/2.0".format(dbricks_url)

# Request Authentication
dbricks_auth = {
    "Authorization": "Bearer {}".format(adbToken),
    "X-Databricks-Azure-SP-Management-Token": azToken,
    "X-Databricks-Azure-Workspace-Resource-Id": ("/subscriptions/{}/resourceGroups/{}/providers/Microsoft.Databricks/workspaces/{}".format(subscription_id, resource_group, databricks_workspace) )
    }

# Creating a databricks token
payload = {
    "lifetime_seconds": 3600, # the token lifetime
    "comment": "This token is created by API call"
}

data =requests.post(f"{dbricks_api}/token/create", headers=dbricks_auth, json=payload)
dict_content = json.loads(data.content.decode('utf-8'))
token = dict_content.get('token_value')


payload = {
    "scope": "my-databricks-secret-scope",
    "initial_manage_principal": "users"
}
res=requests.post(f"{dbricks_api}/secrets/scopes/create", headers={
    "Authorization": "Bearer {}".format(token),
    }, json=payload)

print(res.status_code)