How to set job permissions from REST API?

Question

I am creating automated cluster jobs on Databricks with a service account as part of an ingestion pipeline. I would like to give permissions to certain users so they can monitor the job's progress (e.g. view the Spark UI, logs and metrics). However I cannot find the related configuration option in the 2.0/jobs/create endpoint. See docs.

Answer 1

Databricks has access_control_list property in the create job API to control the permissions.

The following JSON object gives view permissions to all users in the group users when passed to the create job API.

{
    "name": "<job-name>",
    "email_notifications": {
        "no_alert_for_skipped_runs": false
    },
    "webhook_notifications": {},
    "timeout_seconds": 0,
    "max_concurrent_runs": 1,
    "tasks": [
    
    ],
    "queue": {
        "enabled": true
    },
    "run_as": {
        "user_name": "<YOUR_EMAIL>"
    },
    "access_control_list": [
    {
        "user_name": "<YOUR_EMAIL>",
        "permission_level": "IS_OWNER"
    },
    {
        "group_name": "admins",
        "permission_level": "CAN_MANAGE"
    },
    {
        "group_name": "users",
        "permission_level": "CAN_MANAGE_RUN"
    }
]
}

Answer 2

A bit late to join the party but this can still help someone. In an ideal scenario it should be structurally set via pipeline that also creates the job. The below YAML pipeline can set the desired permission. Suitable with Azure Devops but can be changed accordingly.

trigger:
  none

name: Infra update pipeline

pool: 'ubuntu-linux'   # change this accordingly

jobs:
  - job: 'Set_ACL'
    steps:
      - task: AzureCLI@1
        displayName: 'Grant Job Access'
        inputs:
          azureSubscription: '<service_connection>'
          scriptLocation: 'inlineScript'
          inlineScript: |
            #!/bin/bash
            set -ev

            # Argument parsing
            DATABRICKS_URL=$1
            RESOURCE_GROUP=$2
            DATABRICKS_WORKSPACE_NAME=$3
            LIFETIME=$4 
            JOB_ID=$5

            tenantId=$(az account show --query tenantId -o tsv)
            workspace_id=$(az resource show \
              --resource-type Microsoft.Databricks/workspaces \
              -g "$RESOURCE_GROUP" \
              -n "$DATABRICKS_WORKSPACE_NAME" \
              --query id -o tsv)

            # Get a token for the global Databricks application, the resource id is fixed and never changes.
            bearer_token=$(az account get-access-token --resource <your_DBX_resource_id> -o json | jq -r .accessToken)

            # Get a the SP Management Token
            sp_token=$(az account get-access-token --resource https://management.core.windows.net/ -o json | jq -r .accessToken)
            curl -X PUT https://${DATABRICKS_URL}/api/2.0/permissions/jobs/$JOB_ID \
                -H "Authorization: Bearer $bearer_token" \
                -H "Content-Type: application/json" \
                -H "X-Databricks-Azure-SP-Management-Token:$sp_token" \
                --data '{"access_control_list": [{"user_name": "abc.xyz@woohoo.com","permission_level": "IS_OWNER"}]}'
          arguments: >-
            westeurope.azuredatabricks.net           # DBX workspace URL
            <rg_name>            # resource group
            <dbx_resource_name>    # DBX resource name
            400        # PAT life cycle in seconds
            <job_id>   # dbx job id

Different permissions can be found here

Answer 3

I recently came across the same problem of setting permissions to the databricks job from REST API but the api currently not supported this feature. But there is a preview available and its not released officially. url: https://baseuri/api/2.0/preview/permissions/jobs/jobid http: patch jobid: your job id body: {"access_control_list" : [{"user_name" : "" || "group_name" : "" || "service_principal_name" : "","permission_level" : "}] }

Answer 4

I managed to find to find an undocumented 2.0/jobs/reset-acl endpoint. This can be used to assign permissions to an existing job

curl -X POST https://db-cluster/api/2.0/jobs/reset-acl \
    -H "Authorization: Bearer $DATABRICKS_API_TOKEN" \
    -H "Content-Type: application/json" \
    --data '{ "job_id": 123, "grants":[{"user_id": 2345, "permission": "MANAGE_RUN"}, {"user_id": 3456, "permission": "IS_OWNER"}]}'

As this endpoint is not documented anywhere, I'd refrain from using it. However, the only other option I could find is to set it from the UI manually, which is not an acceptable work around for our use case.