Long running-time script in PHP causes NGINX server to get very busy

Question

I'll try to be very specific on this - it won't be easy, so please try to follow.

We have a script that runs with PHP on NGINX - PHP-fpm FastCGI. This script gets information from the user trying to access it, and runs some algorithm on real-time. It cannot be a scheduled process running in the background. Sometimes, it even takes for the page between 5-12 seconds to load - and it's ok. Generally, we collect data from the user and make several outgoing request to third-party servers, collect the data, analyse it and return a response for the user .

The problem is, There are many users running this script, and the server gets very busy - since they're all active connection on the server, waiting for a response. We have 2 servers running under 1 load balancer, and that's not enough. Sometimes the servers have more the 1,500 active connections at a time. You can imagine how these servers respond at that timeframe.

I'm looking for a solution. We can add more and more servers to the LB, but it just sounds absurd that it's the only solution there is. We ran over that script and optimized it to the maximum, I can promise you that - There is no real solution for the long-time running of that script, since it depends on 3rd party servers that take time to respond to us on live traffic.

Is there a solution you can think of, to keep this script as it is - but somehow to lower the impact of these active connection on the overall servers' functioning? Sometimes, they just simply stop to respond.

Thank you very much for reading!

Answer 1

This is a very old question, but since I had a similar problem I can share my solution. Long running scripts impact various parts of the system and cause stresses in webservers (in active connects), php-fpm and mysql/other databases. These tend to cause a number of knock on effects such as other requests starting to fail.

Firstly make sure you have netdata (https://github.com/netdata/netdata) installed on the server. If you are running many instances you might find having a Grafana/Prometheus setup is worth it too.

Next make sure it can see the PHP FPM process, Mysql and Nginx. There are many many things Netdata shows, but for this problem, my key metrics were:

• Connections (mysql_local.connections) - is the database full of connections
• PHP-FPM Active Connections (phpfpm_local.connections) - is PHP failing to keep up
• PHP-FPM Request Duration (phpfpm_local.request_duration) - Is the time to process going thru the roof?
• Disk Utilization Time (disk_util.sda) - this shows if the disk cannot keep up (100% = bad under load)
• Users Open Files (users.files)

Make sure that you have sufficient file handles (https://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/), and the disk is not fully occupied. Both of these will block you from making stuff work, so make them big on the troublesome server.

Next check Nginx has enough resources in nginx.conf:

worker_processes auto;
worker_rlimit_nofile 30000;

events {
    worker_connections 768;
}

This will give you time to work out what is going wrong.

Next look at php-fpm (/etc/php/7.2/fpm/pool.d/www.conf):

• set pm.max_spare_servers high such as 100
• set pm.max_requests = 500 - just in case you have a script that doesn't free itself properly.

Then watch. The problem for me was every request blocks an incoming connection. More requests to the same script will block more connections. The machine can be operating fine, but a single slow script doing a curl hit or a slow SQL statement will take that connection for its entire duration, so 30 seconds = 1 less php process to handle incoming requests. Eventually you hit 500, and you run out. If you can increase the number of FPM processes to match the frequency of slow script requests to the number seconds they run for. So if the script takes 2 seconds, and gets hit 2 times a second, you will need a constant 4 additional fpm worker threads chewed up doing nothing.

If you can do that, stop there - the extra effort beyond that is probably not worth it. If it still feels like it will be an issue - create a second php-fpm instance on the box and send all requests for the slow script to that new instance. This allows you to fail those requests discretely in the case of too much run time. This will give you the power to do two important things:

• Control the amount of resources devoted to the slow script
• Mean that all other scripts are never blocked by the slow script and (assuming the OS limits are high enough) never affected by resource limits.

Hope that helps someone struggling under load!

Answer 2

3 months old question, I know, but I cant help it thinking that:

1. In case you're sure that the sum of the network work for all requests to the third-party servers plus the corresponding processing of the responses inside your PHP script is much lower than the limits of your hardware.

2. Your PHP script is then likely inefficiently busy-looping until all responses come back from the third-party servers

If I were dealing with such an issue I'd do:

1. Stop using your custom external C++ curl thing, as the PHP script is busy-waiting for it anyways.

2. Google and read up on non-busy-looping usage of PHP's curl-multi implementation

Hope this makes sense.

Answer 3

My advice is to set limited timeouts for requests and to use asynchronous requests for each third-party request.

For example, for your page you have to display results of 5 third-party requests. It means, that inside script you call cURL or file_get_contents 5 times, but script becomes frozen for each timeout from third party. Step by step. It means, that if for each request you have to wait 10 seconds for the response - you'll have 50 seconds in total.
User calls the script -> script wait to end -> server is loaded for 50 seconds

Now, if each request to third party will be sent asynchronously - it will reduce script's load time to the maximum request delay. So, you'll have few smaller scripts, that will live shorter life - and it will decrease load on the server.
User calls the script -> script is loaded -> requests are sent -> there are no scripts that are waiting for the response and consuming resources of your server

May the AJAX be with you! ;)