CODEWITHSUNDEEP
azureazure-data-lakeazure-cli
benabderrahmane
benabderrahmane

Reputation: 65

Can we use Azure CLI to upload files to Azure Data Lake Storage Gen2

All I want to do, is to upload files from on prime to Azure Data Lake Storage Gen2 using the Azure CLI (via ` command), but have a connection error! Can I use Azure CLI to to that? Or I have to use another tool? PS: I cannot use Azure Data Factory, I want my job running from my on prime and not from the cloud! Thks.

azure.datalake.store.exceptions.DatalakeRESTException:  HTTP error: 
ConnectionError(MaxRetryError("HTTPSConnectionPool(host='storageAccount.azuredatalakestore.net', port=443): 
Max retries exceeded with url: /webhdfs/v1/my-file-system/data.csv?OP=GETFILESTATUS&api-version=2018-05-01 
(Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fc7ed169c50>: 
Failed to establish a new connection: [Errno -2] Name or service not known')

Upvotes: 1

Views: 2793

Answers (2)

benabderrahmane
benabderrahmane

Reputation: 65

Got it work ^^ So the problem was with my authentication method, to make it work, you have to add your user as a Data Lake Storage Contributor + Owner. For anyone looking for the Role in the UI it's called "Storage Blob Data Contributor (Preview)". For a Resource Group choose Access Control (IAM) | Add in the blade locate the role Storage Blob Data Contributor (Preview) and assign access to the Users, Groups or Roles as meets your needs.

Upvotes: 1

Joy Wang
Joy Wang

Reputation: 42043

No, Azure CLI for gen2 filesystem is not available, see this link.

enter image description here

If you want to upload file, here are two workarounds for you to refer.

  1. Use Azure Storage Explorer

  2. Use AzCopy v10, note only v10 supports Azure Data Lake Storage Gen2 APIs. Use myaccount.dfs.core.windows.net as a URI to call the ADLS Gen2 APIs.

Upvotes: 3

Related Questions