CODEWITHSUNDEEP
azure-databricksjobsazure-service-principaldatabricks-workflows
Rio
Rio

Reputation: 398

GET request to databricks jobs API with Azure Token, Incorrect Claim Exception

I am using the jobs API to get the status of the databricks job created by azure service principal. I create the azure token using the code below

import requests
import json
from databricks_cli.sdk.service import JobsService
from databricks_cli.sdk.api_client import ApiClient
import time 

#generating toekn request for service principal
sp_id = dbutils.secrets.get("scope name", "key")
sp_sec = dbutils.secrets.get("scope name", "key")
token_endpoint = "https://login.microsoftonline.com/<databricks-instance>/oauth2/token"
token_request_data = {
    "grant_type":"client_credentials",
    "client_id": sp_id,
    "client_secret": sp_sec,
    "scope" :"2ff814a6-3304-4ab8-85cb-cd0e6f879c1d%2F.default"
}

#send the request to get the token
response = requests.post(token_endpoint, data= token_request_data)
token_sp= response.json()["access_token"]

Then I can use this token successfully to run the databricks job. However when I want to get the status of the job using the same token, I get this error:

#send the request using run ID to get the state of the job with service principal token 

uri = "https://adb-1366553829746932.12.azuredatabricks.net/api/2.1/jobs/runs/get?run_id=831336"
headers = {
             "Authorization":f"Bearer {token_sp}",
             "Content-Type":"application/json",
             }

response=requests.get(uri,headers = headers)
response.text

'<html>\n<head>\n<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>\n<title>Error 400 io.jsonwebtoken.IncorrectClaimException: Expected aud claim to be: 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d, but was: 00000002-0000-0000-c000-000000000000.</title>\n</head>\n<body><h2>HTTP ERROR 400</h2>\n<p>Problem accessing /api/2.1/jobs/runs/get. Reason:\n<pre>    io.jsonwebtoken.IncorrectClaimException: Expected aud claim to be: 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d, but was: 00000002-0000-0000-c000-000000000000.</pre></p>\n</body>\n</html>\n'

I can get the job status if I use the PAT instead of service principal token.

I am not sure why this token works for POST request but not the GET request.

Upvotes: 1

Views: 2646

Answers (2)

Alex Ott
Alex Ott

Reputation: 87259

The problem was in the not completely correct URL for the AAD token generation service. Per documentation, it should be /oauth2/v2.0/token, not /oauth2/token (this is old version that doesn't use scope, but need to receive resource id). You also need to change 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d%2F.default to 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default

Here is the working code:

import requests
import json
import time 

#generating toekn request for service principal
sp_id = "..."
sp_sec = dbutils.secrets.get("scope", "secret")
tenant_id = "..."

token_endpoint = f"https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/token"
token_request_data = {
    "grant_type":"client_credentials",
    "client_id": sp_id,
    "client_secret": sp_sec,
    "scope" :"2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default"
}
response = requests.post(token_endpoint, data= token_request_data)
token_sp= response.json()["access_token"]

And then we can use it:

headers = { 'Authorization': f'Bearer {token_sp}', 'Content-Type': 'application/json' } 
url = "https://adb-....azuredatabricks.net/api/2.0/clusters/list"
response = requests.get(url, headers=headers)
response.text

and it gives normal list of clusters:

Out[10]: {'clusters': [{'cluster_id': '...',
   'creator_user_name': '....',
   'driver': {'private_ip': '10.1.4.5',
    'node_id': '78368a91af7a4b4f8d4e97f65790b7c2',
    'instance_id': 'f7ce71746e1c4d1696817f5ead0765b5',
    'start_timestamp': 1682060511817,
    'node_attributes': {'is_spot': False},

Upvotes: 1

Vamsi Bitra
Vamsi Bitra

Reputation: 2764

I reproduce the same thing in my environment and successfully ran the job as per the above requirement with status.

enter image description here

Modified your code as per the above requirement, generated a bearer token, set up authentication and perform GET request to get job run status

import requests
import json

# set up authentication
token = 'dapi54bxxxxxxxxxxxaf75c2536acd6eaa'
headers = {'Authorization': f'Bearer {token}'}

run_id1 = '55xx'
endpoint = f'/api/2.0/jobs/runs/get?run_id={run_id1}'

#GET request to get job run status
resp = requests.get(f'https://adb-8411xxxxxxx.13.azuredatabricks.net{endpoint}', headers=headers)

# Check the response status code
if resp.status_code == 200:
    resp_json = json.loads(resp.text)
    job1_status = response_json['state']['life_cycle_state']
    job1_result = response_json['state']['result_state']
    
    # Print the job run status and result state
    print(f'Job run status: {job1_status}')
    print(f'Job run result: {job1_result}')
    
    if job1_result == 'SUCCESS':
        print('Job Completed Successfully.')
    elif job1_result == 'FAILED':
        print('Job Encountered an Error.')
else:
    print(f'Error: {resp.status_code} - {resp.text}')

Successfully executed the code with Job run status:

enter image description here

Upvotes: 0

Related Questions