CODEWITHSUNDEEP
powershellpowershell-5.0
joebegborg07
joebegborg07

Reputation: 839

"OutOfMemoryException" when using Compress-Archive with 60Gb of files

I'm trying to compress IIS Log files (60GB total) using Compress-Archive, however I get the error:

"Exception of type 'System.OutOfMemoryException' was thrown."

$exclude = Get-ChildItem -Path . | Sort-Object -Descending | Select-Object -First 1
$ArchiveContents = Get-ChildItem -Path . -Exclude $exclude | Sort-Object -Descending
Compress-Archive -Path $ArchiveContents -DestinationPath .\W3SVC2.zip -Force

I've already adjusted MaxMemoryPerShellMB to 2048MB and restarted the WinRM service.

RAM Memory consumption exceeds 6GB when the command is executed.

Upvotes: 4

Views: 7633

Answers (2)

0xFK
0xFK

Reputation: 2728

The below solution provide enhanced performance better than Compress-Archive

Create OptimalCompression.ps1

$source = $args[0]
$destination = $args[1]
Write-host "Compressing $source to $destination"
$compressionLevel = [System.IO.Compression.CompressionLevel]::Optimal
$includeBaseDirectory = $false
Add-Type -AssemblyName System.IO.Compression.FileSystem
[System.IO.Compression.ZipFile]::CreateFromDirectory($source,$destination,$compressionLevel,$includeBaseDirectory)

Run the Command with required parameters

C:\> powershell .\OptimalCompress.ps1 c:\YourSourceFolder c:\YourDestinationFile.zip

Upvotes: 1

TheMadTechnician
TheMadTechnician

Reputation: 36322

As suggested by Matt, I would recommend archiving it into sets of files. Something like:

For($i=0;$i -le $ArchiveContents.Count;$i=$i+10){
    Compress-Archive -Path $ArchiveContents[$i..($i+9)] -DestinationPath .\W3SVC2-$i.zip -Force
}

That way you're only working with 10 files at a time. You may want to adjust the number of files, depending on how many you have, and how large they are, ie: If you have 300 files at 200MB each then 10 at a time may be good, whereas if you have 3000 files at 20MB each you may want to increase that to 50 or even 100.

Edit: I just looked, and Compress-Archive supports adding files to an archive by specifying the -Update parameter. You should be able to do the same thing as above, slightly modified, to make 1 large archive.

For($i=0;$i -le $ArchiveContents.Count;$i=$i+10){
    Compress-Archive -Path $ArchiveContents[$i..($i+9)] -DestinationPath .\W3SVC2.zip -Update
}

That should just add 10 files at a time to the target archive, rather than trying to add all of the files at once.

Upvotes: 3

Related Questions