CODEWITHSUNDEEP
powershelldatabricksazure-databricksazure-data-lake-gen2
Learn2Code
Learn2Code

Reputation: 147

how to delete subfolder in ADLS gen2 via azure databricks

I need to delete ADLS Gen2 subfolder with specific name using databricks dbutils but not able to perform wild card recursion.

e.g. 
adlsstorageaccount/container1/folder2/folder3/folderA/**abc**/.parquet
adlsstorageaccount/container1/folder2/folder3/folderB/**abc**/.parquet
adlsstorageaccount/container1/folder2/folder3/folderC/**abc**/.parquet

Need to delete subfolder name "abc" & its contents only, entire path is dynamic.

Upvotes: 0

Views: 1422

Answers (2)

Hubert Dudek
Hubert Dudek

Reputation: 1730

you need just use magic command %sh in databricks and remove it with wilcard using standard linux commands, something like:

%sh
rm /dbfs/mnt/adlsstorageaccount/container1/folder2/folder3/folderA/*abc*/*.parquet

Upvotes: 1

SwethaKandikonda
SwethaKandikonda

Reputation: 8244

We have reproduced the same Folders in our environment and here are the commands that worked.

$FileSystemName="<Your Container Name>"
$dirname="folder2/folder3/folderA/abc/"
$ctx=New-AzStorageContext -StorageAccountName '<Your Storage Account>' -StorageAccountKey '<Your Access Key>'
Remove-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $dirname

and then confirm it with Y - 'yes'

Here are the screenshots for your reference

Before using the command execution enter image description here

After the command execution enter image description here

REFERENCES: https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-directory-file-acl-powershell#delete-a-directory

Upvotes: 1

Related Questions