Getting stack trace from Perl "Out of memory" error

Question

tl;dr: How to dump a perl stack trace when a Perl httpd process runs out of memory.

We've got a mod_perl 2 server, Perl 5.8.8, RHEL 5.6, Linux 2.6.18.

Very occasionally and unpredictably, a child httpd process starts using up all available memory at an alarming rate. We've at least used BSD::Resource::setrlimit(RLIMIT_VMEM, ...), so that the process dies with "Out of memory" before bringing down the server.

We don't know where in the code this is happening, and it's difficult to reproduce without hours of load testing.

What we'd really like is a way to get a Perl stack trace just before the process runs out of memory, so we know what code is causing this. Unfortunately, "Out of memory" is an untrappable error.

Here are the options I'm considering, each with their drawbacks:

1) Use the $^M emergency memory pool. Requires us to recompile perl with -DPERL_EMERGENCY_SBRK and -Dusemymalloc.

2) Put in tons of log statements, then analyze the logs to see where the process is stopping short.

3) Write an external script that constantly scans the pool of httpd processes, and if it sees one using a lot of memory, sends it a USR2 signal (which we've arranged to dump a stack trace).

4) Somehow have the process monitor its own memory continuously, and dump a stack trace when memory gets high but before the "Out of Memory" error.

Thanks!

Jon

Answer 1

According to Practical mod_perl, adding Apache::Debug level => 4; to your code will cause it to allocate the $^M emergency pool and call Carp::confess().

https://books.google.co.vi/books?id=i3Ww_7a2Ff4C&pg=PT699&lpg=PT699&dq=how+do+i+get+a+stack+trace+when+perl+dies+with+an+out+of+memory+error&source=bl&ots=Id7ll9Gng2&sig=r8SpZ4jvFfjzfM2eSmoV2DjWLnY&hl=en&sa=X&ved=0ahUKEwiwqNitporbAhXow1kKHZo9CRoQ6AEIODAC#v=onepage&q=how%20do%20i%20get%20a%20stack%20trace%20when%20perl%20dies%20with%20an%20out%20of%20memory%20error&f=false

Answer 2

You can get a backtrace with mod_backtrace, see Andy Millar's introduction. The backtrace is on the C level, so you either need

• a bit of Perl internals knowledge to infer the Perl stack by simply looking at the backtrace or
• to run gdb, set a breakpoint at the crashy function and examine the Perl stack and lexical variables with the gdb macros from the mod_perl book.

Answer 3

You might try using LD_PRELOAD to load a custom version of malloc/free. It should not be necessary to relink or recompile anything. Just set LD_PRELOAD to have a .so that has the same interface as malloc, and this preloaded version of malloc will be loaded instead of the normal system malloc when you run your executable. For instance, this is a common strategy for detecting memory errors with efence.

I don't think efence will work for you here as that is for detecting memory overwrites, not debugging out of memory (I don't know what it does on OOM). I think what you might want to check out failmalloc. I have never used this one but it sounds like it might do what you want (I only skimmed the front page).