apache httpd using up mem until hang

Question

It appears we might have a (growing) memory issue in our apache httpd somewhere.

Quick pic: http://screencast.com/t/NGAzarD87O

Notice that it's been running fine on the original (physical) server for a while. Now on the new VM (with more memory and cpu), it runs but slowly eats away at mem/swap until the system hangs.

If I restart the httpd the mem jumps back up (if we catch it).

httpd.x86_64                           2.2.3-76.el5_9         installed
PHP 5.1.6 (cli) (built: Jun 22 2012 06:20:25)
MySQL Server version: 5.0.95

I don't think it can be any of the scripts that are run/etc as they ran fine for years on the physical machine. We've tried to match all configs (http, php, etc) on new machine but can't figure out why httpd keeps growing.

$ ps -ylC httpd --sort:rss
S   UID   PID  PPID  C PRI  NI   RSS    SZ WCHAN  TTY          TIME CMD
S     0 13814     1  0  78   0 29208 68382 -      ?        00:00:00 httpd
S    48 20854 13814  0  76   0 34876 70930 semtim ?        00:00:00 httpd
S    48 20853 13814  0  75   0 36592 71387 semtim ?        00:00:00 httpd
S    48 13822 13814  0  75   0 36780 71430 semtim ?        00:00:00 httpd
S    48 20696 13814  0  75   0 37092 71520 semtim ?        00:00:00 httpd
S    48 13821 13814  0  75   0 37184 71529 semtim ?        00:00:01 httpd
S    48 13820 13814  0  75   0 37220 71527 -      ?        00:00:01 httpd
S    48 13824 13814  0  75   0 37236 71513 semtim ?        00:00:01 httpd
S    48 13818 13814  0  75   0 37636 71547 semtim ?        00:00:01 httpd
S    48 13819 13814  0  75   0 37636 71617 semtim ?        00:00:01 httpd
S    48 13823 13814  0  75   0 37888 71689 semtim ?        00:00:01 httpd
S    48 13825 13814  0  75   0 37900 71676 semtim ?        00:00:01 httpd

UPDATE: while writing this question out (maybe 10-15 minutes) I reran the above and the RSS are all sitting at ~51072 instead of the above at ~37000

Here's a run from the python program

 Private  +   Shared  =  RAM used       Program
.... SNIPPED OUT ....
208.0 MiB +  25.5 MiB = 233.5 MiB       httpd (12)
---------------------------------
                        477.1 MiB

PHP mem settings:

max_execution_time = 30 
max_input_time = 60  
memory_limit = 152M  

Modules:

$ apachectl -M
httpd: Could not reliably determine the server's fully qualified domain name, using 192.168.1.2 for ServerName
Loaded Modules:
 core_module (static)
 mpm_prefork_module (static)
 http_module (static)
 so_module (static)
 auth_basic_module (shared)
 auth_digest_module (shared)
 authn_file_module (shared)
 authn_alias_module (shared)
 authn_anon_module (shared)
 authn_dbm_module (shared)
 authn_default_module (shared)
 authz_host_module (shared)
 authz_user_module (shared)
 authz_owner_module (shared)
 authz_groupfile_module (shared)
 authz_dbm_module (shared)
 authz_default_module (shared)
 ldap_module (shared)
 authnz_ldap_module (shared)
 include_module (shared)
 log_config_module (shared)
 logio_module (shared)
 env_module (shared)
 ext_filter_module (shared)
 mime_magic_module (shared)
 expires_module (shared)
 deflate_module (shared)
 headers_module (shared)
 usertrack_module (shared)
 setenvif_module (shared)
 mime_module (shared)
 dav_module (shared)
 status_module (shared)
 autoindex_module (shared)
 info_module (shared)
 dav_fs_module (shared)
 vhost_alias_module (shared)
 negotiation_module (shared)
 dir_module (shared)
 actions_module (shared)
 speling_module (shared)
 userdir_module (shared)
 alias_module (shared)
 rewrite_module (shared)
 proxy_module (shared)
 proxy_balancer_module (shared)
 proxy_ftp_module (shared)
 proxy_http_module (shared)
 proxy_connect_module (shared)
 cache_module (shared)
 suexec_module (shared)
 disk_cache_module (shared)
 file_cache_module (shared)
 mem_cache_module (shared)
 cgi_module (shared)
 version_module (shared)
 perl_module (shared)
 php5_module (shared)
 proxy_ajp_module (shared)
 python_module (shared)
 ssl_module (shared)
Syntax OK

httpd.conf settings:

IfModule prefork.c>  ignore format on these tags
StartServers       8
MinSpareServers    5
MaxSpareServers   20
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  4000

# worker MPM
StartServers         2
MaxClients         150
MinSpareThreads     25
MaxSpareThreads     75
ThreadsPerChild     25
MaxRequestsPerChild  0

Using the program at https://code.google.com/p/check-httpd-limits/wiki/Documentation I've gotten:

Check Apache Httpd MPM Config Limits (Version 2.4)
by Jean-Sebastien Morisset - http://surniaulula.com/

Httpd Binary

 - CONFIG                : /etc/httpd/conf/httpd.conf
 - EXE                   : /usr/sbin/httpd
 - MPM                   : prefork
 - ROOT                  : /etc/httpd
 - VERSION               : 2.2

Httpd Processes

 - PID 10860 (httpd)     :  106.93 MB /   3.95 MB shared
 - PID 13814 (httpd)     :   28.52 MB /   6.36 MB shared [excluded from averages]
 - PID 13818 (httpd)     :  180.28 MB /   4.29 MB shared
 - PID 13819 (httpd)     :  182.67 MB /   4.04 MB shared
 - PID 13820 (httpd)     :  182.45 MB /   4.08 MB shared
 - PID 13821 (httpd)     :  185.53 MB /   4.04 MB shared
 - PID 13822 (httpd)     :  176.12 MB /   4.36 MB shared
 - PID 13823 (httpd)     :  180.05 MB /   4.04 MB shared
 - PID 13824 (httpd)     :  182.21 MB /   4.05 MB shared
 - PID 13825 (httpd)     :  179.36 MB /   4.04 MB shared
 - PID 20696 (httpd)     :  180.10 MB /   4.04 MB shared
 - PID 20853 (httpd)     :  180.39 MB /   4.03 MB shared
 - PID 20854 (httpd)     :  180.79 MB /   4.04 MB shared
 - PID 21003 (httpd)     :  159.77 MB /   4.05 MB shared

 - HttpdRealAvg          :  166.09 MB [excludes shared]
 - HttpdSharedAvg        :    4.05 MB
 - HttpdRealTot          : 2225.76 MB [excludes shared]
 - HttpdRunning          :      14

Httpd Config

 - StartServers          : 8
 - ServerLimit           : 256
 - MinSpareServers       : 5
 - MaxSpareServers       : 20
 - MaxRequestsPerChild   : 4000
 - MaxClients            : 256

Server Memory

 - Cached                :   671.46 MB
 - MemFree               :   547.88 MB
 - MemTotal              :  3819.89 MB
 - SwapFree              :  5951.89 MB
 - SwapTotal             :  5951.99 MB

Calculations Summary

 - OtherProcsMem         :   370.74 MB (MemTotal - Cached - MemFree - HttpdRealTot - HttpdSharedAvg)
 - FreeMemNoHttpd        :  3449.15 MB (MemFree + Cached + HttpdRealTot + HttpdSharedAvg)
 - MaxLimitHttpdMem      : 42523.09 MB (HttpdRealAvg * MaxClients + HttpdSharedAvg)
 - AllProcsTotalMem      : 42893.83 MB (OtherProcsMem + MaxLimitHttpdMem)

Maximum Values for MemTotal (3819.89 MB)

   
        StartServers               8    # (no change) Default is 5
        ServerLimit               21    # (256 -> 21) MaxClients
        MinSpareServers            5    # (no change) Default is 5
        MaxSpareServers           20    # (no change) Default is 10
        MaxRequestsPerChild     4000    # (no change) Default is 10000
        MaxClients                21    # (256 -> 21) (MemFree + Cached + HttpdRealTot + HttpdSharedAvg) / HttpdRealAvg
   

Result

ERROR: AllProcsTotalMem (42893.83 MB) exceeds MemTotal (3819.89 MB) and free swap (5951.89 MB) by 33122.05 MB.

Answer 1

Sometimes allocating too much memory creates a memory problem. Unless there is a specific reason to allocate 152M to PHP, that's high. Drop that down to 32M or 64M. If a specific script required more, enable it for that script only.

Next, check your database allocation. Often memory is the first thing people allocate to make something run faster when there are other items that should be done (like query tuning).

Essentially, too much allocation creates a backup of information for the I/O. Sounds backwards, I know. I had a similar issue and it was over allocation of resources. Drop your resources to their default values and see if this changes the memory usage at all. Tune from there to find the best settings.

Answer 2

There can be any number of reasons Apache could take too much memory. Though there is no universal "answer" there is a troubleshooting list:

1. Re-install Apache and all modules.
2. Disable all modules you don't need. In addition to potentially using extra resources, these can also be security hazards.
   1. mod_autoindex will allow anybody to view all files in any diretory in your public_html directory so long as there is no index.* in that directory.
   2. mod_alias will will not only reveal that you're running Apache, but will also be a good indicator you haven't removed extra modules, and will make you a more likely target.
   3. Remove all proxy modules unless you're actually using them.
   4. Remove all mod_cgi, mod_php, etc that you don't need. The more file types you allow Apache to execute, the more ways there are for a malicious person to be malicous.
3. Check your server configurations. Don't be afraid to research what certain directives do, and don't be afraid to create a backup of your apache.conf and change a few things. The worst case is that Apache will fail to start, or Apache will crash.