CODEWITHSUNDEEP
azure-blob-storagedatabricksazure-databricks
Krzysztof
Krzysztof

Reputation: 103

Azure Databricks: error 403 while doing dbutils.fs.ls on mounted directory

I have an Azure Databricks workspace, a storage account with hierarchical namespace enabled and a service principal. I have mounted the storage successfully (the output was "True"):

configs = {
    "fs.azure.account.auth.type": "OAuth",
    "fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
    "fs.azure.account.oauth2.client.id": "[redacted]",
    "fs.azure.account.oauth2.client.secret": "[redacted]",
    "fs.azure.account.oauth2.client.endpoint": "https://login.microsoftonline.com/[redacted]/oauth2/token"
}

dbutils.fs.mount(
  source = "abfss://[redacted]@[redacted].dfs.core.windows.net/",
  mount_point = "/mnt/demo",
  extra_configs = configs
)

Now I try to view the mounted directory contents:

dbutils.fs.ls("/mnt/demo")

and I get error:

Operation failed: "This request is not authorized to perform this operation using this permission.", 403, GET, https://[redacted].dfs.core.windows.net/[redacted]?upn=false&resource=filesystem&maxResults=5000&timeout=90&recursive=false, AuthorizationPermissionMismatch, "This request is not authorized to perform this operation using this permission.

I have double checked that my service principal has Storage Blob Data Contributor permissions to the storage account.

What am I doing wrong? Any help will be much appreciated.

Upvotes: 0

Views: 226

Answers (2)

Krzysztof
Krzysztof

Reputation: 103

Thanks everyone for useful tips! The problem, however, was more prosaic. It turned out that I had two Service Principals with the same name. In my code have used Client ID/Secret of first one, but I have granted Blob Contributor role to another one.

Upvotes: 0

Dileep Raj Narayan Thumula
Dileep Raj Narayan Thumula

Reputation: 3250

As you mentioned you have provided the SPN the Storage Blob contributor role.

I have tried mounting the ADLS and used the below it did work for me:

dbutils.fs.ls("/mnt/new_raw")

Results:

[FileInfo(path='dbfs:/mnt/new_raw/Delta_folder/', name='Delta_folder/', size=0, modificationTime=1726556613000),
 FileInfo(path='dbfs:/mnt/new_raw/Parquet_folder/', name='Parquet_folder/', size=0, modificationTime=1726567040000),
 FileInfo(path='dbfs:/mnt/new_raw/control_table/', name='control_table/', size=0, modificationTime=1726556615000),
 FileInfo(path='dbfs:/mnt/new_raw/synapse/', name='synapse/', size=0, modificationTime=1726031781000)]

When I munted the ADLS I have followed the below steps:

  • I have Provided My KEYVAULT Key vault adminstartor to SPN & Azuredatabricks application.
  • I have provided the storage blob contributor role to the azuredatabricks application
  • Also please check if the storage has been protected by firewall. If the storage account is storage has been protected by firewall you can add the outbound IP to the whitelist of storage.

enter image description here

enter image description here

If you are using the Unity catalog enabled workspace refer to Create a storage credential for connecting to Azure Data Lake Storage Gen2

Upvotes: 1

Related Questions