Reputation: 565
Problems with Apache servers and A LOT of httpd processes
We have a problem with our Apache server. It's really slow and cpu usage is really high (sometimes showing a constant 100%).
It's on a ubuntu system.
Any ideas what might be causing this?
"top" shows this (there's probably even more httpd processes that didn't fit in the terminal window):
top - 11:33:19 up 1:07, 2 users, load average: 37.79, 35.90, 29.14
Tasks: 86 total, 10 running, 76 sleeping, 0 stopped, 0 zombieCpu(s): 90.6%us, 9.4%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 786432k total, 379584k used, 406848k free, 0k buffersSwap: 0k total, 0k used, 0k free, 0k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5507 nobody 16 0 54540 19m 5204 S 4 2.6 0:03.70 httpd 5653 nobody 17 0 59996 24m 4808 R 4 3.2 0:01.98 httpd
3277 nobody 15 0 201m 51m 6724 R 2 6.7 2:14.30 mysqld 5260 nobody 15 0 53740 18m 4972 S 2 2.4 0:06.53 httpd
5333 nobody 15 0 54536 19m 5152 S 2 2.5 0:06.50 httpd 3923 nobody 15 0 52644 18m 6372 S 1 2.4 0:16.80 httpd
5519 nobody 15 0 51252 15m 4792 S 1 2.0 0:03.02 httpd 4048 nobody 15 0 52808 17m 5040 R 1 2.3 0:13.86 httpd
5240 nobody 15 0 53980 18m 4916 S 1 2.4 0:08.42 httpd
5249 nobody 15 0 52908 17m 5188 S 1 2.3 0:08.23 httpd 5385 nobody 15 0 52568 16m 4824 S 1 2.2 0:04.19 httpd
5395 nobody 16 0 52536 16m 4804 S 1 2.1 0:03.90 httpd 5422 nobody 16 0 51408 15m 4888 S 1 2.1 0:03.49 httpd
5503 nobody 15 0 52536 16m 4800 S 1 2.1 0:02.67 httpd 3646 nobody 15 0 52572 16m 5012 S 1 2.2 0:25.42 httpd
3948 nobody 15 0 52844 17m 5008 R 1 2.3 0:17.15 httpd 5283 nobody 15 0 54000 18m 4944 S 1 2.4 0:07.05 httpd
5300 nobody 15 0 54748 19m 5276 S 1 2.6 0:06.20 httpd 5304 nobody 15 0 52876 17m 4904 S 1 2.2 0:06.45 httpd
5305 nobody 15 0 52812 17m 4884 S 1 2.2 0:06.02 httpd 5400 nobody 16 0 52060 16m 4800 R 1 2.1 0:04.10 httpd
5423 nobody 15 0 54764 19m 5196 S 1 2.6 0:04.37 httpd 5433 nobody 15 0 52168 16m 4844 R 1 2.1 0:03.38 httpd
5497 nobody 15 0 52572 16m 4820 S 1 2.2 0:02.83 httpd 5498 nobody 15 0 53076 17m 4864 S 1 2.3 0:03.14 httpd
5502 nobody 15 0 51260 15m 4804 S 1 2.0 0:02.70 httpd 5505 nobody 15 0 54920 19m 4976 S 1 2.5 0:02.68 httpd
5546 nobody 15 0 52584 16m 4664 S 1 2.2 0:02.48 httpd 5550 nobody 15 0 51244 15m 4796 S 1 2.0 0:02.42 httpd
5652 nobody 15 0 51976 17m 5912 S 1 2.3 0:01.24 httpd 5659 nobody 15 0 55012 19m 4980 S 1 2.6 0:01.25 httpd
5707 nobody 15 0 51416 15m 4880 S 1 2.1 0:00.09 httpd 5710 nobody 15 0 51240 15m 4792 S 1 2.0 0:00.14 httpd
5399 nobody 15 0 52568 16m 4856 S 1 2.2 0:03.86 httpd 5435 nobody 15 0 52596 16m 4864 S 1 2.2 0:02.96 httpd
5467 nobody 15 0 52316 16m 4856 R 1 2.1 0:02.93 httpd 5501 nobody 15 0 52956 17m 4844 S 1 2.2 0:02.79 httpd
5525 nobody 15 0 54508 19m 4916 S 1 2.5 0:03.41 httpd 5547 nobody 15 0 52604 18m 5708 S 1 2.3 0:02.77 httpd
5590 root 15 0 19208 1360 1044 R 1 0.2 0:00.39 top 5627 nobody 15 0 55648 20m 5164 S 1 2.6 0:02.56 httpd
5629 nobody 15 0 51248 15m 4792 S 1 2.0 0:01.62 httpd 5647 nobody 15 0 52956 17m 4884 S 1 2.2 0:01.16 httpd
5656 nobody 15 0 52588 17m 4884 S 1 2.2 0:00.94 httpd 5665 nobody 15 0 51244 15m 4792 S 1 2.0 0:01.36 httpd
5709 nobody 15 0 51928 15m 4628 S 1 2.1 0:00.02 httpd 5711 nobody 16 0 51148 15m 4600 S 1 2.0 0:00.02 httpd
Upvotes: 23
Views: 65370
Answers (3)
Reputation: 1940
System and Httpd Information
System:# cat /proc/version
Linux version 3.10.0-514.2.2.el7.x86_64 ([email protected]) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Tue Dec 6 23:06:41 UTC 2016
# httpd -v
Server version: Apache/2.4.6 (CentOS)
Server built: Nov 14 2016 18:04:44
Make MPM Prefork Configure
Get MPM In The Default# sudo apachectl -t -D DUMP_MODULES |grep mpm
[Thu Jan 12 16:06:11.778080 2017] [so:warn] [pid 23727] AH01574: module wsgi_module is already loaded, skipping
mpm_prefork_module (shared)
That means mpm_prefork_module is installed and enabled. You can change it to the mode of event, worker or prefork(default) in the file 00-mpm.conf, which file path is /etc/httpd/conf.modules.d.
The reference of Apache MPM is Apache MPM Common Directives
Add Personal Configure# sudo cat >> /etc/httpd/conf/httpd.conf <<-'EOF'
<IfModule prefork.c>
ServerLimit 20
StartServers 15
MinSpareServers 5
MaxSpareServers 10
MaxClients 15
MaxRequestsPerChild 25
</IfModule>
EOF
- My VM has 2GB memory and 2 Processors
Read details of this configure in the above-mentioned reference.
Restart Httpd
# sudo systemctl restart httpd.service
Inspect Status
# systemctl status httpd.service
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2017-01-12 13:40:12 CST; 2h 48min ago
Docs: man:httpd(8)
man:apachectl(8)
Main PID: 16576 (httpd)
Status: "Total requests: 3625; Current requests/sec: -70.2; Current traffic: 0 B/sec"
Memory: 869.7M
CGroup: /system.slice/httpd.service
├─16576 /usr/sbin/httpd -DFOREGROUND
├─24122 /usr/sbin/httpd -DFOREGROUND
├─24125 /usr/sbin/httpd -DFOREGROUND
├─24154 /usr/sbin/httpd -DFOREGROUND
├─24157 /usr/sbin/httpd -DFOREGROUND
├─24211 /usr/sbin/httpd -DFOREGROUND
├─24219 /usr/sbin/httpd -DFOREGROUND
└─24236 /usr/sbin/httpd -DFOREGROUND
Jan 12 13:40:12 192-168-1-68.node systemd[1]: Starting The Apache HTTP Server...
Jan 12 13:40:12 192-168-1-68.node httpd[16576]: [Thu Jan 12 13:40:12.614941 2017] [so:warn] [pid 16576] AH01574: module wsgi_module is already loaded, skipping
Jan 12 13:40:12 192-168-1-68.node systemd[1]: Started The Apache HTTP Server.
or
# ps aux|grep httpd
root 16576 0.0 0.0 232232 504 ? Ss 13:40 0:01 /usr/sbin/httpd -DFOREGROUND
apache 24296 10.9 4.8 542936 91500 ? S 16:29 0:16 /usr/sbin/httpd -DFOREGROUND
apache 24310 10.6 4.7 542804 89752 ? S 16:29 0:15 /usr/sbin/httpd -DFOREGROUND
apache 24343 11.2 6.5 542804 123944 ? S 16:29 0:15 /usr/sbin/httpd -DFOREGROUND
apache 24344 11.1 6.5 542804 124084 ? S 16:29 0:14 /usr/sbin/httpd -DFOREGROUND
apache 24345 11.6 6.7 542804 126616 ? S 16:29 0:15 /usr/sbin/httpd -DFOREGROUND
apache 24347 11.0 6.6 542804 125764 ? S 16:29 0:14 /usr/sbin/httpd -DFOREGROUND
apache 24377 12.0 6.7 542804 126396 ? S 16:29 0:15 /usr/sbin/httpd -DFOREGROUND
apache 24378 12.4 6.7 542804 126448 ? S 16:29 0:15 /usr/sbin/httpd -DFOREGROUND
apache 24408 12.3 6.6 542804 126032 ? S 16:29 0:14 /usr/sbin/httpd -DFOREGROUND
apache 24450 18.6 5.4 511560 102520 ? R 16:30 0:10 /usr/sbin/httpd -DFOREGROUND
apache 24458 19.4 4.7 498476 89548 ? S 16:30 0:08 /usr/sbin/httpd -DFOREGROUND
apache 24466 21.4 4.5 495424 86416 ? R 16:31 0:07 /usr/sbin/httpd -DFOREGROUND
apache 24484 0.5 0.3 236456 5760 ? S 16:31 0:00 /usr/sbin/httpd -DFOREGROUND
root 24496 0.0 0.0 112652 964 pts/0 S+ 16:31 0:00 grep --color=auto httpd
My apache servers get a request at 2 per second. Before, the numbers of httpd processes increased until the system halted. Aftering setting the MPM Prefork, it works fine.
Upvotes: 3
Reputation: 634
In my experience, the dozens of httpd processes were DOS attacks. After installing mod_evasive, the dozens of httpd processes showing via top were all but eliminated and my mysql CPU load went from an average 100%+ / 99% of the time utilization down to an expected 20% load during heavier query operations.
To install mod_evasive, on RHEL (CentOS/Fedora/Redhat) systems, do:
yum install -y mod_evasive
Restart Apache
/sbin/service httpd restart
or
systemctl restart httpd
and viola.
Even if a DOS attack isn't your issue, mitigating attacks before they happen is wise.
Upvotes: 14
Reputation: 11949
Apache needs to fork a process to handle each connected client. Read more about that here.
The upshot is that you have lots of processes because you have lots of connections. If your server is running slowly as a result then here are a few options:
- Optimise your application: profile your application's code and try and optimise the areas that place the most load on the server.
- Consider changing to a lower impact web server, such as nginx. This could increase the number of connections that your server can handle.
- Add page caching, using something like varnish
- Add another server, and load balance between them
- Increase the capabilities of the existing server (e.g. increase RAM, upgrade processor, install a faster hard disk).
All of these options have a cost, in time or money. Unfortunately that's just the way it is :)
The quickest wins in my opinion would probably be options 2 and 3. But it means downtime while they're being installed, or having a backup server to work on.
Upvotes: 32
Related Questions
- Apache http server problems
- Why Apache spawns too much processes?
- Too many '/usr/sbin/apache2 -k start' processes
- Httpd Process High memory usage
- Httpd processes bring CPU down (99% usage)
- To many long running Apache processes in READ status without requests after a certain time
- How can I prevent tons of apache processes spawning when I start apache and proceeding to kill my machine?
- Number of apache processes
- Centos server crash with too many httpd process in D state
- server problems - strange apache requests