CODEWITHSUNDEEP
nginxphp
Bian Jiaping
Bian Jiaping

Reputation: 966

php-fpm: ERROR: [pool ] no free scoreboard slot

Recently my php application is often unavailable. It happens several times a day.

In php-fpm error log I see these messages:

[23-Mar-2015 16:49:42] NOTICE: fpm is running, pid 8038
[23-Mar-2015 16:49:43] NOTICE: ready to handle connections
[23-Mar-2015 16:49:47] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 98 idle, and 105 total children
[23-Mar-2015 16:49:57] WARNING: [pool www] child 8132 exited on signal 11 (SIGSEGV) after 13.640528 seconds from start
[23-Mar-2015 16:49:57] NOTICE: [pool www] child 8151 started
[23-Mar-2015 16:49:59] WARNING: [pool www] child 8091 exited on signal 11 (SIGSEGV) after 16.331350 seconds from start
[23-Mar-2015 16:49:59] NOTICE: [pool www] child 8155 started

[24-Mar-2015 09:38:01] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 15 total children
[24-Mar-2015 09:38:01] ERROR: [pool ] no free scoreboard slot

[24-Mar-2015 17:00:39] NOTICE: [pool www] child 27622 started
[24-Mar-2015 17:00:51] WARNING: [pool www] child 27482 exited on signal 11 (SIGSEGV) after 230.000581 seconds from start
[24-Mar-2015 17:00:51] NOTICE: [pool www] child 27623 started
[24-Mar-2015 17:01:07] WARNING: [pool www] child 27522 exited on signal 11 (SIGSEGV) after 246.737464 seconds from start
[24-Mar-2015 17:01:07] ERROR: [pool ] no free scoreboard slot
[24-Mar-2015 17:01:13] WARNING: [pool www] child 27538 exited on signal 11 (SIGSEGV) after 252.268165 seconds from start
[24-Mar-2015 17:01:13] ERROR: [pool ] no free scoreboard slot

Executing service php-fpm reload solves the problem temporarily, and it raises again after sometime.

According to nginx's access log, there are at most 1000 requests per minute, and about 600 requests are passed to php-fpm(others are static files). Seems not heavy.

Server Environment:

php-fpm configuration:

pm = dynamic
pm.max_children = 100
pm.start_servers = 100
pm.min_spare_servers = 100
pm.max_spare_servers = 1000
pm.max_requests = 500

I have tried pm = static and changed other params, but neither did work.

As a temporary solution, I write a shell script to check service availability regularly. If not available, reload php-fpm.

But how could I solve the problem completely?

Upvotes: 2

Views: 2275

Answers (1)

Bian Jiaping
Bian Jiaping

Reputation: 966

Finally the problem disappeared after I modified one line of terrible code that tried to fetch all records(sometimes up to 10K) of a table from MySQL.

I checked nginx's access log, and was surprised to find that many requests to an api yielded 500 response code.

After some digging, I found it's caused by one line of code that tried to fetch all records of a table from MySQL. It's okay when there aren't many records, but when there are more than 10K records, the request ended with 500 internal error.

After modifying the terrible code, no more requests to that api yield 500, and the php-fpm error disappears, too.

2015-12-28 update:

I think the real reason that breaks down the entire application is that the requests to that badly-implemented api occupy too much memory, and need too much execution time.

As a result, php-fpm worker pool is filled with such requests, and therefore there is no workers to handle other requests. So the entire application breaks down.

The reason that those requests failed with 500 may be that the memory is runned out, or the execution time exceeds max_execution_time specified in php.ini.

If you run into this problem, check your access log(nginx or php-fpm), and try to find the requests that always occupy too much memory/time (memory usage can only be found in php-fpm's access log), or the requests that always failed. Then check whether the problem is caused by those requests.

Upvotes: 1

Related Questions