CODEWITHSUNDEEP
apache-sparkpysparkazure-synapse
paone
paone

Reputation: 937

Write data to Azure Data Lake Storage Gen 2 using Azure Synapse Analytics notebook

I am connecting to a RESTful api using Azure Synapse Analytics notebook and write the json file to Azure Data Lake Storage Gen 2.

pyspark code:

import requests
response = requests.get('https://api.web.com/v1/data.json')
data = response.json()
from pyspark.sql import *
df = spark.read.json(sc.parallelize([data]))
from pyspark.sql.types import *
account_name = "name of account"
container_name = "name of container"
relative_path = "name of file path"    #abfss://<container_name>@<storage_account_name>.dfs.core.windows.net/<path>
adls_path = 'abfss://%s@%s.dfs.core.windows.net/%s' % (container_name, account_name, relative_path)
spark.conf.set('fs.%s@%s.dfs.core.windows.net/%s' % (container_name, account_name), "account_key") #not sure I'm doing the configuration right
df.write.mode("overwrite").json(adls_path)

Error:

Py4JJavaError : An error occurred while calling o536.json.
: Operation failed: "This request is not authorized to perform this operation.", 403, HEAD, https://storageaccount.dfs.core.windows.net/container/?upn=false&action=getAccessControl&timeout=90

Upvotes: 6

Views: 7741

Answers (1)

CHEEKATLAPRADEEP
CHEEKATLAPRADEEP

Reputation: 12788

Note: Storage Blob Data Contributor: Use to grant read/write/delete permissions to Blob storage resources.

If you are not assigning Storage Blob Data Contributor to users who are accessing the storage account, they will be not able to access the data from ADLS gen2 due to the lack of permission on the storage account.

If they try to access data from ADLS gen2 without the "Storage Blob Data Contributor" role on the storage account, they will receive the error message: Operation failed: "This request is not authorized to perform this operation.",403.

Once the storage account is created, select Access control (IAM) from the left navigation. Then assign the following roles or ensure they are already assigned. Assign yourself to the Storage Blob Data Owner role on the Storage Account.

After granting Storage Blob Data Contributor role on the storage account wait for 5-10 minutes and re-try the operation.

enter image description here

Upvotes: 3

Related Questions