CODEWITHSUNDEEP
azureazure-devopsazure-pipelinesazure-databricks
Selena Li
Selena Li

Reputation: 21

Azure DevOps CD Pipeline to Deploy Library to Databricks DBFS 403 Forbidden Error

I'm following the tutorial Continuous integration and delivery on Azure Databricks using Azure DevOps to automate the process to deploy and install library on an Azure Databricks cluster. However, I'm stucked in the step "Deploy the library to DBFS" using task Databricks files to DBFS in Databricks Script Deployment Task extension by Data Thirst.

It continuously gives me this error:

##[error]The remote server returned an error: (403) Forbidden.

The configuration of this task is shown below: configuration of Databricks files to DBFS

I've checked with my token that it works fine when I try to upload the libraries manually through Databricks CLI. Thus, the problem shouldn't be due to the permission of the token.

Can anyone suggest any solution to this? Or is there any alternative way to deploy libraries to clusters on Azure Databricks via the release CD pipelines on Azure DevOps?

Upvotes: 2

Views: 3221

Answers (4)

Anh Nguyen Ngoc
Anh Nguyen Ngoc

Reputation: 135

I think, to address the 403 Forbidden error you're encountering when deploying libraries to DBFS on Azure Databricks through Azure DevOps, you will have to ensure your setup in the Databricks Script Deployment Task by Data Thirst is correctly configured with the right Databricks workspace URL and personal access token (PAT). A common oversight is a mismatch or typo in these configurations. e.g., ensure your Databricks workspace URL is correctly formatted (e.g., https://.azuredatabricks.net) and your PAT is accurately inputted in the task's settings. Here's an illustrative example of how you might configure a task in YAML for an Azure DevOps pipeline, emphasizing the need to securely store and reference the PAT:

- task: DataThirstLtd.databricksDeployScripts.databricksDeployment@1
  inputs:
    DatabricksInstance: 'https://<region>.azuredatabricks.net'
    PersonalAccessToken: '$(DatabricksToken)'
    LocalPath: '$(System.DefaultWorkingDirectory)/path/to/your/library'
    DbfsPath: '/dbfs/path/to/destination'

IMPORTANT: $(DatabricksToken) should be a secure variable or secret that stores your Databricks personal access token.

Upvotes: 0

Vivek Atal
Vivek Atal

Reputation: 533

I also faced similar problem while using the Databricks Script Deployment Task created by Data Thirst. Then switched to DevOps for Azure Databricks created by Microsoft DevLabs. Below are the steps I used to work with Databricks CLI to achieve what I wanted to do as part of Azure Release Pipeline:

  1. First, added Use Python version task. Referred to Python 3.7
  2. Then, added Configure Databricks CLI. Provided workspace URL, e.g. adb-1234567890123456.12.azuredatabricks.net, and provided the personal access token by referring to a secret variable
  3. Added a Command Line Script task, and added Databricks CLI scripts as inline code. Moreover, added --profile AZDO along with the scripts as this profile is configured in the previous step. E.g., dbfs cp $(System.DefaultWorkingDirectory)/abcd dbfs:/mytempfiles --recursive --overwrite --profile AZDO

Upvotes: 0

Cristian
Cristian

Reputation: 11

Did you check your Azure Region in Databricks? If you don't use the same Azure Region in Azure Devops, you will get 403 error.

Upvotes: 1

Selena Li
Selena Li

Reputation: 21

After trying multiple times, it turns out if you don't use the extension and use Databricks CLI in the pipeline to directly upload files, the uploading will work smoothly. Hope this helps if someone got the same problem.

Upvotes: 0

Related Questions