CODEWITHSUNDEEP
azureparquetazure-powershell
Raksha Hegde
Raksha Hegde

Reputation: 35

Download parquet file from ADL Gen2 using powershell

I am using below powershell command to download TSV file from ADL to local system and it works absolutely fine

#this appid has access to ADL
[string] $AppID  = "bbb88818-aaaa-44fb-q2345678901y" 
 [string] $TenantId  = "ttt88888-xxxx-yyyy-q2345678901y"
 [string] $SubscriptionName  = "Sub Sample"
 [string] $AzureDataLakeAccountName  = "sample.blob.core.windows.net"

 [string] $AzureDataLakeSrcFilePath  = "/accounts/sample/test.tsv"
 [string] $LocalTargetFilePathName  = "D:\MoveToModern"


Write-Host "AppID = " $AppID
Write-Host "TenantId = " $TenantId
Write-Host "SubscriptionName = " $SubscriptionName
Write-Host "AzureDataLakeAccountName = " AzureDataLakeAccountName
Write-Host "AzureDataLakeSrcFilePath = " $AzureDataLakeSrcFilePath
Write-Host "LocalTargetFilePathName = " $LocalTargetFilePathName



#this is the access key of the appid
$AccessKeyValue = "1234567=u-r.testabcdefaORYsw5AN5"

$azurePassword    = ConvertTo-SecureString $AccessKeyValue -AsPlainText -Force
$psCred           = New-Object System.Management.Automation.PSCredential($AppID, $azurePassword)
Login-AzureRmAccount -Credential $psCred -ServicePrincipal -Tenant $TenantId

Get-AzureRmSubscription

Get-AzureRmSubscription -SubscriptionName $SubscriptionName  | Set-AzureRmContext
Export-AzureRmDataLakeStoreItem -AccountName $AzureDataLakeAccountName -Path $AzureDataLakeSrcFilePath  -Destination $LocalTargetFilePathName -Force

But when I replace sourcefile path with value /accounts/sample/test-V4.parquet, I get below error:

ADLSException: Error in getting metadata for path /accounts/Partner/Non-PII/Account/Account-V4.parquet. Operation: GETFILESTATUS failed with HttpStatus:Forbidden Error: Unexpected error in JSON parsing of the error stream. Content-Type of error response: application/xml. ExceptionType: Newtonsoft.Json.JsonReaderException ExceptionMessage: Unexpected character encountered while parsing value: <. Path '', line 0, position 0. AuthenticationFailedServer failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. RequestId:de2e4b16-001e-002e-625f-c92785000000

enter image description here

Can you please let me know how can I download parquet file from ADL Gen2 using powershell.

Upvotes: 0

Views: 758

Answers (1)

Jim Xu
Jim Xu

Reputation: 23111

According to the information you provide, you do not have enough permissions to do that. So you get Forbidden error. If you want to download a file from Azure data lake store with a service principal, we need to grant the security principal read access to the file and give the security principal Execute permissions to the container and each folder in the hierarchy of folders that lead to the file.

For example enter image description here

Please check it and set the right ACL. Regarding how to do that, please refer to here

If you want to download files from Azure Data Lake Gen2, I suggest you use PowerShell module Az.Storage. For more details, please refer to here

For example

  1. Configure ACL for the service principal.

If you want to download a file from Azure data lake Gen2 with a service principal, we need to grant the security principal read access to the file and give the security principal Execute permissions to the container and each folder in the hierarchy of folders that lead to the file. Regarding how to configure it, please refer to here.

  1. Script
$AppID  = "" 
$AccessKeyValue  = ""
$TenantId=""


$azurePassword    = ConvertTo-SecureString $AccessKeyValue -AsPlainText -Force
$psCred           = New-Object System.Management.Automation.PSCredential($AppID, $azurePassword)
Connect-AzAccount -Credential $psCred -ServicePrincipal -Tenant $TenantId


$AzureDataLakeAccountName  = ""
$ctx =New-AzStorageContext -StorageAccountName $AzureDataLakeAccountName -UseConnectedAccount 

$filesystemName="<your container name>"
$path="<your blob name>"
$LocalTargetFilePathName  = ""
Get-AzDataLakeGen2ItemContent -Context $ctx -FileSystem $filesystemName -Path $path -Destination $LocalTargetFilePathName

Upvotes: 0

Related Questions