CODEWITHSUNDEEP
powershellget-childitem
Jasin
Jasin

Reputation: 425

Delete massive amount of files without running out of memory

There is COTS app we have that creates reports and never deletes it. So we need to start cleaning it up. I started doing a foreach and would run out of memory on the server (36GB) when it got up to 50ish million files. After searching it seemed you could change it like so

Get-ChildItem -path $Path -recurse | foreach {

and it won't go through memory but process each item at a time. I can get to 140 million files before I run out of memory.

Clear-Host
#Set Age to look for
$TimeLimit = (Get-Date).AddMonths(-4)
$Path = "D:\CC\LocalStorage"

$TotalFileCount = 0
$TotalDeletedCount = 0

Get-ChildItem -Path $Path -Recurse | foreach {
    if ($_.LastWriteTime -le $TimeLimit) {
        $TotalDeletedCount += 1
        $_.Delete()
    }
    $TotalFileCount += 1

    $FileDiv = $TotalFileCount % 10000
    if ($FileDiv -eq 0 -and $TotalFileCount -ne 0) {
        $TF = [string]::Format('{0:N0}', $TotalFileCount)
        $TD = [string]::Format('{0:N0}', $TotalDeletedCount)
        Write-Host "Files Scanned : " -ForegroundColor Green  -NoNewline
        Write-Host "$TF" -ForegroundColor Yellow    -NoNewline
        Write-Host " Deleted: " -ForegroundColor Green  -NoNewline
        Write-Host "$TD" -ForegroundColor Yellow  
    }

Is there a better way to do this? My only next thought was not to use the -Recurse command but make my own function that calls itself for each directory.

EDIT:

I used the code provided in the first answer and it does not solve the issue. Memory is still growing.

$limit = (Get-Date).Date.AddMonths(-3)
$totalcount  = 0
$deletecount = 0
$Path = "D:\CC\"

Get-ChildItem -Path $Path -Recurse -File |  Where-Object { $_.LastWriteTime -lt $limit } |  Remove-Item -Force

Upvotes: 2

Views: 1459

Answers (1)

Ansgar Wiechers
Ansgar Wiechers

Reputation: 200273

Using the ForEach-Object and the pipeline should actually prevent the code from running out of memory. If you're still getting OOM exceptions I suspect that you're doing something in your code that counters this effect, which you didn't tell us about.

With that said, you should be able to clean up your data directory with something like this:

$limit = (Get-Date).Date.AddMonths(-4)

Get-ChildItem -Path $Path -Recurse -File |
  Where-Object { $_.LastWriteTime -lt $limit } |
  Remove-Item -Force -WhatIf

Remove the -WhatIf switch after you verified that everything is working.

If you need the total file count and the number of deleted files, add counters like this:

$totalcount  = 0
$deletecount = 0

Get-ChildItem -Path $Path -Recurse -File |
  ForEach-Object { $totalcount++; $_ } |
  Where-Object { $_.LastWriteTime -lt $limit } |
  ForEach-Object { $deletecount++; $_ } |
  Remove-Item -Force -WhatIf

I don't recommend printing status information to the console when you're bulk-processing large numbers of files. The output could significantly slow down the processing. If you must have that information, write it to a log file and tail that file separately.

Upvotes: 4

Related Questions