CODEWITHSUNDEEP
azurepowershellazure-storageazure-storage-accountazure-storage-files
Bala
Bala

Reputation: 1

How to remove azure file share old data from the azure storage account?

I have 3 months of old data which is stored on azure storage account ? Now i want remove the data if it >= 30days enter image description here

Upvotes: 0

Views: 2252

Answers (1)

kavya Saraboju
kavya Saraboju

Reputation: 10859

The following script lists files/FileDir recursively in a file share and delete the files older than 14 days.You may give the required day limit. Refered from thread here.

Refer this for the best practices before delete activity.

$ctx = New-AzStorageContext -StorageAccountName $accountName -StorageAccountKey $key
 $shareName = <shareName>
    
 $DirIndex = 0
 $dirsToList = New-Object System.Collections.Generic.List[System.Object]
    
 # Get share root Dir
 $shareroot = Get-AzStorageFile -ShareName $shareName -Path . -context $ctx 
 $dirsToList += $shareroot 
    
 # List files recursively and remove file older than 14 days 
 While ($dirsToList.Count -gt $DirIndex)
 {
     $dir = $dirsToList[$DirIndex]
     $DirIndex ++
     $fileListItems = $dir | Get-AzStorageFile
     $dirsListOut = $fileListItems | where {$_.GetType().Name -eq "AzureStorageFileDirectory"}
     $dirsToList += $dirsListOut
     $files = $fileListItems | where {$_.GetType().Name -eq "AzureStorageFile"}
    
     foreach($file in $files)
     {
         # Fetch Attributes of each file and output
         $task = $file.CloudFile.FetchAttributesAsync()
         $task.Wait()
    
         # remove file if it's older than 14 days.
         if ($file.CloudFile.Properties.LastModified -lt (Get-Date).AddDays(-14))
         {
             ## print the file LMT
             # $file | Select @{ Name = "Uri"; Expression = { $_.CloudFile.SnapshotQualifiedUri} }, @{ Name = "LastModified"; Expression = { $_.CloudFile.Properties.LastModified } } 
    
             # remove file
             $file | Remove-AzStorageFile
         }
     }
     #Debug log
     # Write-Host  $DirIndex $dirsToList.Length  $dir.CloudFileDirectory.SnapshotQualifiedUri.ToString() 
 }

(OR)

Delete activity can be configured from Azure data Factory(ADF) by following the steps below.This require linking your azure storage account with the ADF by giving account name, fileshare name and path if needed.

  • Deploy an ADF if not already configured:

f1

  • Open ADF and create a PipeLine with any name.

f2

The Process is :We get the metadata of the file shares in storage account selected>Loop through them>configure Delete activity for the files inside them older than 30 days(or say some x days)

    1. Search for get meta data >> select and drag to place in the area shown and name the file. 2) Then navigate to Dataset(This data set points to the files in storage account),select new . So select azure file storage>Name it 3) >>Select the format as csv 4)>>Link your account by setting up the properties > give the file path.

f3

  • To get files older than or equals 30 days you can configure endtime as @adddays(utcnow(),-30)

enter image description here

  • Now we have to use foreach loop to iterate through the array of files. Drag and drop foreach loop from activities and connect/link the arrwoed line with get metadata.

f5

  • In settings ,Tick mark sequential box to iterate files in sequential order. childItems is the property of getmetadata output JSON which has array of objects . So select Dynamic content for Items and configure as @activity("Get old files").output.childItems and click on finish. ( Here Get old files is the name of my get activity created previously )

f6

  • In the Foreach activity ,edit the activities and set a delete activity as below.

enter image description here enter image description here

  • Go to the dataset where we previously linked our file storage account.

f9

  • Create a file name parameter and link it to delete activity by adding dynamic content for file path by selecting that filepath as @dataset().FileName

f10

  • Then go to the delete pipeline and add the file name as shown.

enter image description here

  • You may also add logging setting to link account and see the activity happening after the path is debugged.

enter image description here

Other reference: clean-up-files-by-built-in-delete-activity-in-azure-data-factory/

Upvotes: 1

Related Questions