Authorization error on my storage account when lisitng files from databricks

Question

I have the strange issue where I dont understand why Im having the authorization error:

Im running this code with out any error:

dbutils.fs.ls("abfss://bronze@mycontainer.dfs.core.windows.net/")

it lists all the folders in there:

[FileInfo(path='abfss://bronze@mycontainer.dfs.core.windows.net/graph_api/', name='graph_api/', size=0, modificationTime=1737733983000),
 FileInfo(path='abfss://bronze@mycontainer.dfs.core.windows.net/manual_tables/', name='manual_tables/', size=0, modificationTime=1737734175000),
 FileInfo(path='abfss://bronze@mycontainer.dfs.core.windows.net/process_logging/', name='process_logging/', size=0, modificationTime=1737734175000)
]

But when I try to do :

dbutils.fs.ls("abfss://bronze@mycontainer.dfs.core.windows.net/graph_api/")

An error occurred while calling o426.ls. : Operation failed: "This request is not authorized to perform this operation.", 403, GET

I have the external location that has the credential (pointing to accesConnector of the workspace, which is Storage blob data contributor on my storage account) attached to it. I am the owner of both. Im aslo storage blob data contributor myself on storage account.

Im facing same issue when I do dbutils.fs.put

EDIT:

I think its netowrking issue? not sure BUT when I Enabled from all networks it let me list of the files inside the folder.

Infra setup: I have the Vnet inject databricks, and my Storage account has Enabled from selected virtual networks and IP addresses those two subnets are whitelisted. Each subnet has the Service endpoint of Storage account attached. I dont use the private endpoint for storage account.

How can I fix the issue?

Answer 1

An error occurred while calling o426.ls. : Operation failed: "This request is not authorized to perform this operation.", 403, GET

The ERROR you are facing seems to be related to network restrictions and access permissions for your Azure Data Lake Storage Gen2 account.

As you mentioned that

I think its networking issue? not sure BUT when I Enabled from all networks it let me list of the files inside the folder.

You need a separate private endpoint for each storage resource you want to access, including Blobs, Data Lake Storage Gen2, Files, Queues, Tables, or Static Websites. On the private endpoint, these storage services are specified as the target sub-resource of the linked storage account (privatelink.dfs.core.windows.net)

Reference: Use private endpoints for Azure Storage

Also try the below

I faced a similar issue with my Gen2 storage account, which contained two filesystems and one standard container.

To resolve it, I assigned the Storage Blob Data Contributor role to the service principal app, which granted access to the mounted containers from Databricks.

Here is how you can assign permissions to the service principal app:

Open your Storage Account Navigate to IAM (Access Control) Click Add → Add role assignment Search for and select Storage Blob Data Contributor Under Members, choose your SPN

This should grant the necessary permissions for Databricks access.