CODEWITHSUNDEEP
c#powershelltobase64string
abnerkanezzer
abnerkanezzer

Reputation: 97

Base64 Encode Large Zip File Using Powershell

I am trying to base64 encode a ~66MB zip file to a string and write it to a file using Powershell. I'm working with a limitation that ultimately I have to include the base64 encoded file string directly into a Powershell script, such that when the script is run at a different location, the zip file can be recreated from it. I'm not limited to using Powershell to create the base64 encoded string. It's just what I'm most familiar with.

The code I'm currently using:

$file = 'C:\zipfile.zip'
$filebytes = Get-Content $file -Encoding byte
$fileBytesBase64 = [System.Convert]::ToBase64String($filebytes)
$fileBytesBase64 | Out-File 'C:\base64encodedString.txt'

Previously, the files I have worked with have been small enough that encoding was relatively fast. However, I am now finding that the file I'm encoding results in the process eating up all my RAM and ultimately is untenably slow. I get the feeling that there's a better way to do this, and would appreciate any suggestion.

Upvotes: 5

Views: 5971

Answers (2)

filimonic
filimonic

Reputation: 4634

UPDATE 2023-08-17 for big files, memory usage and speed

As @JohnRanger mentioned, there is a problem with previous answer that it has source file limit to ~1.5 GiB and memory-consumable.

Solution is to use file streams and CryptoStream(... ToBase64Transform...)

$sourcePath = "C:\test\windows_11.iso"
$targetPath = "C:\test\windows_11.iso.b64"
$size = Get-Item -Path $sourcePath | Select -ExpandProperty Length
$stopwatch = [System.Diagnostics.Stopwatch]::StartNew()

$converterStream = [System.Security.Cryptography.CryptoStream]::new(
    [System.IO.File]::OpenRead($sourcePath),
    [System.Security.Cryptography.ToBase64Transform]::new(), 
    [System.Security.Cryptography.CryptoStreamMode]::Read,
    $false) # keepOpen = $false => When we close $converterStream, it will close source file stream also
$targetFileStream = [System.IO.File]::Create($targetPath)
$converterStream.CopyTo($targetFileStream)
$converterStream.Close() # And it also closes source file stream because of keepOpen = $false parameter.
$targetFileStream.Close() # Flush() is called internally.

$stopwatch.Stop()
Write-Host "Elapsed: $($stopwatch.Elapsed.TotalSeconds) seconds for $([Math]::Round($size / 1MB))-Mbyte file"

⚠️ This code runs over 30 times faster on PS7 compared to PS5

PS7: It takes 3.1 seconds for 5316 MiB file on my machine.

File copying with [IO.File]::Copy(...) takes 1.8 seconds.

PS5: It takes 115 seconds on my machine. I played with buffer size in [System.IO.File]::Create(...) and $converterStream.CopyTo(...), but did not get any reasonable performance difference : for 5316 MiB I had worst result of 115 seconds for default buffer sizes and best 112 seconds while playing buffer sizes. Maybe for slow targets buffer sizes will have more impact (i.e. working with slow disk or network share). Performance always hit single core of CPU.

File copying with [IO.File]::Copy() takes 1.8 seconds.

Upvotes: 2

filimonic
filimonic

Reputation: 4634

⚠️ UPDATE 2023-08-17: There is a better solution.

Im my case it takes less than 1 second to encode or decode 117-Mb file.

Src file size: 117.22 MiB
Tgt file size: 156.3 MiB
Decoded size: 117.22 MiB
Encoding time: 0.294
Decoding time: 0.708

Code I making measures:

$pathSrc = 'D:\blend5\scena31.blend'
$pathTgt = 'D:\blend5\scena31.blend.b64'
$encoding = [System.Text.Encoding]::ASCII

$bytes = [System.IO.File]::ReadAllBytes($pathSrc)
Write-Host "Src file size: $([Math]::Round($bytes.Count / 1Mb,2)) MiB"
$swEncode = [System.Diagnostics.Stopwatch]::StartNew()
$B64String = [System.Convert]::ToBase64String($bytes, [System.Base64FormattingOptions]::None)
$swEncode.Stop()
[System.IO.File]::WriteAllText($pathTgt, $B64String, $encoding)

$B64String = [System.IO.File]::ReadAllText($pathTgt, $encoding)
Write-Host "Tgt file size: $([Math]::Round($B64String.Length / 1Mb,2)) MiB"
$swDecode = [System.Diagnostics.Stopwatch]::StartNew()
$bytes = [System.Convert]::FromBase64String($B64String)
$swDecode.Stop()
Write-Host "Decoded size: $([Math]::Round($bytes.Count / 1Mb,2)) MiB"

Write-Host "Encoding time: $([Math]::Round($swEncode.Elapsed.TotalSeconds,3)) s"
Write-Host "Decoding time: $([Math]::Round($swDecode.Elapsed.TotalSeconds,3)) s"

Upvotes: 3

Related Questions