Operation failed: "This request is not authorized to perform this operation." in Synapse with a Pyspark Notebook

Question

I try to execute the following command line:

mssparkutils.fs.ls("abfss://mycontainer@myadfs.dfs.core.windows.net/myfolder/")

I get the error:

Py4JJavaError: An error occurred while calling z:mssparkutils.fs.ls.
: java.nio.file.AccessDeniedException: Operation failed: "This request is not authorized to perform this operation.", 403, GET, https://myadfs.dfs.core.windows.net/mycontainer?upn=false&resource=filesystem&maxResults=5000&directory=myfolder&timeout=90&recursive=false, AuthorizationFailure, "This request is not authorized to perform this operation.

I followed the steps described in this link

by granting access to me and my Synapse workspace the role of "Storage Blob Data Contributor" in the container or file system level:

Even that, I still get this persistent error. Am I missing other steps?

Answer 1

Another possibility is that you have Storage Firewall running on the Storage Account containing your Data Lake/Blob Container.

As far as I can make out there is not way around this unless you have private networking enabled on your Synapse instance - in which case you can create a Managed Private Endpoint to enable a Spark to Storage network route.

Answer 2

I got the same kind of error in my environment. I just followed this official document and done the repro, now it's working fine for me. You can follow the below code it will solve your problem.

Sample code:

from pyspark.sql import SparkSession
account_name = 'your_blob_name'
container_name = 'your_container_name'
relative_path = 'your_folder path'
linked_service_name = 'Your_linked_service_name'

sas_token = mssparkutils.credentials.getConnectionStringOrCreds(linked_service_name)

Access to Blob Storage

path = 'wasbs://%s@%s.blob.core.windows.net/%s' % (container_name,account_name,relative_path)   
spark.conf.set('fs.azure.sas.%s.%s.blob.core.windows.net' % (container_name,account_name),sas_token)
  
print('Remote blob path: ' + path)

Sample output:

Updated answer

Reference to configure Spark in pyspark notebook:

https://techcommunity.microsoft.com/t5/azure-synapse-analytics-blog/notebook-this-request-is-not-authorized-to-perform-this/ba-p/1712566