CODEWITHSUNDEEP
powershell
Maumee River
Maumee River

Reputation: 263

Powershell crashing due to memory constants

I've got a server running around 500 powershell processes. Each of these processes are designed to make WMI calls across our environment. I've been careful to verify that I do not use up all of the server's available memory or CPU. When I have all 500 processes running, I'm at around 70% memory usage.

Just in case anybody is wondering how the individual processes are handled, they are executed using a gearman job worker. Basically a shell python script that calls a powershell script...times 500.

The issue i'm running into is that some of my powershell processes are crashing after running a few hours.

Some of the errors that I'm getting are: 

A new guard page for the stack cannot be created

When I open event viewer, I see these events when processes crash 

Fault bucket , type 0
Event Name: PowerShell
Response: Not available
Cab Id: 0

Problem signature:
P1: powershell.exe
P2: 6.3.9600.16394
P3: System.OutOfMemoryException
P4: System.OutOfMemoryException
P5: oft.PowerShell.ConsoleHost.ReportExceptionFallback
P6: lization.EncodingTable.nativeCreateOpenFileMapping
P7: Consol.. main thread
P8: 
P9: 
P10: 

Attached files:

These files may be available here:
C:\path

Analysis symbol: 
Rechecking for solution: 0
Report Id: ID
Report Status: 2048
Hashed bucket:

I'm guessing it has something to do with powershell running out of memory, but the server is not peaked, and not all processes crash, it is sporadic.

Any help would be appreciated.

Here are more crash results, the powershell fault module names are different from time to time: 

  Problem Event Name:   APPCRASH
  Application Name: powershell.exe
  Application Version:  6.3.9600.16384
  Application Timestamp:    52158733
  Fault Module Name:    ntdll.dll
  Fault Module Version: 6.3.9600.16408
  Fault Module Timestamp:   523d45fa
  Exception Code:   c00000fd
  Exception Offset: 00069abb
  OS Version:   6.3.9600.2.0.0.272.7
  Locale ID:    1033
  Additional Information 1: 624b
  Additional Information 2: 624b484d3cf74536f98239c741379147
  Additional Information 3: a901
  Additional Information 4: a901f876e92d1eb79eb3a513defef0c6

  Problem signature:
  Problem Event Name:   APPCRASH
  Application Name: powershell.exe
  Application Version:  6.3.9600.16384
  Application Timestamp:    52158733
  Fault Module Name:    combase.dll
  Fault Module Version: 6.3.9600.16408
  Fault Module Timestamp:   523d3001
  Exception Code:   c00000fd
  Exception Offset: 0001a360
  OS Version:   6.3.9600.2.0.0.272.7
  Locale ID:    1033
  Additional Information 1: 81ca
  Additional Information 2: 81cae32566783b059420874b47802c3e
  Additional Information 3: b637
  Additional Information 4: b6375e6f6a866fc9d00393d4649231b8

Upvotes: 4

Views: 2900

Answers (4)

Maumee River
Maumee River

Reputation: 263

Thanks everyone for the responses, it turns out that I had a memory leak in my powershell code that was causing memory usage to spike every now and then. Since I was not watching the server at every second, I missed when the memory usage spiked.

An interesting note, it appears that Powershell will not use more then 80% of available memory on a server before killing its own processes.

I had to increase the available memory to 56GB and now I'm not running into any issues whatsoever. I've been running 600 powershell processes for a week now and have not had one crash on me.

Upvotes: 0

egrunin
egrunin

Reputation: 25073

If you're using TaskManager to check on memory usage, you might try Process Explorer instead. It sometimes gives very different results.

Upvotes: 0

2174714
2174714

Reputation: 288

Doesn't .Net have a limit of memory?

Upvotes: 0

NickBuckley
NickBuckley

Reputation: 77

have you looked at your max memory allocation per shell?

get-item WSMan:\localhost\Shell\MaxMemoryPerShellMB

and if its too low changing this;

set-item WSMan:\localhost\Shell\MaxMemoryPerShellMB 2048

Upvotes: 1

Related Questions