CODEWITHSUNDEEP
powershell
Nish87
Nish87

Reputation: 29

OutOfMemoryException when running my PowerShell script

Here's the PowerShell script I am using to add "segment99" to the beginning of all the text files (one by one) within a folder:

Set Environmental Variables:

$PathData = '<<ESB_Data_Share_HSH>>\RwdPnP'

Go to each text file in the specified folder and add header to the file:

Get-ChildItem $PathData -filter 'test_export.txt'|%{

$content = '"segment99" ' + [io.file]::ReadAllText($_.FullName)
[io.file]::WriteAllText(($_.FullName -replace '\.txt$','_99.txt'),$content)

}

This is giving me the following error:

Error: Exception calling "ReadAllText" with "1" argument(s): "Exception of type 'Syste
Error: m.OutOfMemoryException' was thrown."
Error: At D:\apps\MVPSI\JAMS\Agent\Temp\JAMSTemp13142.ps1:17 char:51
Error: + $content = '"segment99" ' + [io.file]::ReadAllText <<<< ($_.FullName)
Error:     + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
Error:     + FullyQualifiedErrorId : DotNetMethodException
Error:

I am running this code on a folder that has 20 files, each over 2 GB.

How can I fix this?

Upvotes: 3

Views: 3467

Answers (2)

jon Z
jon Z

Reputation: 16636

Copying a header file + a large file to a new file will be less prone to outofmemory exceptions (for files of that size):

$header = '"segment99"'
$header | out-file header.txt -encoding ASCII
$pathdata = "."
Get-ChildItem $PathData -filter 'test_export.txt' | %{
  $newName = "{0}{1}{2}" -f $_.basename,"_99",$_.extension
  $newPath = join-path (split-path $_.fullname) $newname
  cmd /c copy /b "header.txt"+"$($_.fullname)" "$newpath"
}

Upvotes: 3

Roman Kuzmin
Roman Kuzmin

Reputation: 42063

This is not optimal code but it solves the task without reading all text to memory: it adds the header to the first line and then outputs other lines. Also, note that it does nothing if the input file is empty.

Get-ChildItem $PathData -Filter 'test_export.txt' | %{
    $header = $true
    Get-Content $_.FullName | .{process{
        if ($header) {
            '"segment99" ' + $_
            $header = $false
        }
        else {
            $_
        }
    }} | Set-Content ($_.FullName -replace '\.txt$', '_99.txt')
}

Upvotes: 2

Related Questions