CODEWITHSUNDEEP
azureazure-pipelinesazure-blob-storageazure-data-lake-gen2
Tim
Tim

Reputation: 307

Any way to batch-upload a set of files to ADLS gen2 storage?

I want to replace my blob storage with the ADLS Gen 2 functionality in Azure. Point is, I upload a batch of files in a build pipeline with the following command to a certain container in Blob:

az storage blob upload-batch \
          --destination "storage-container-files" \
          --account-name "accountname" \
          --source "./files"

This works perfectly fine. Now I want to do the same thing but then to Gen2 datalake, for which I have to use the az storage blob .... command. However I cannot find something similar to 'upload-batch', only single file upload commands. I would like not having to loop in the yaml or in a Python script.

Is there any simple way to batch upload like I could with blob?

Upvotes: 0

Views: 990

Answers (2)

Bowman Zhu
Bowman Zhu

Reputation: 7146

Sure, there is:

trigger:
- none

pool:
  vmImage: ubuntu-latest

steps:
- task: AzureCLI@2
  inputs:
    azureSubscription: 'testbowman_in_AAD'
    scriptType: 'bash'
    scriptLocation: 'inlineScript'
    inlineScript: |
      az storage fs directory upload -f xxx/dir --account-name bowmantest -s "files" --recursive --connection-string "xxx"

This is my repo structure:

enter image description here

And successfully:

enter image description here

Refer to this:

https://learn.microsoft.com/en-us/cli/azure/storage/fs/directory?view=azure-cli-latest#az-storage-fs-directory-upload

Upload files or subdirectories to a directory in ADLS Gen2 file system.

Upvotes: 1

VenkateshDodda
VenkateshDodda

Reputation: 5506

We don't have any cmdlet to upload a batch of file (set of files) for ADLS gen2 Account.

We have az storage fs file upload cmdlet to upload a single file only.I would suggest you use AZCOPY.exe to upload bulk of files to your adls gen 2 storage account.

For additional information, you can refer to these below links:

  1. How to upload file to azure storage account using AzCopy.
  2. How to upload bluk of files to adls gen2 account using python.

Upvotes: 0

Related Questions