Load Average/Perfomance problem with php-fpm

Question

When site loaded by users (about 3000 online) I have big perfomance error. I have nginx+php-fpm. When I have some online(~3k) php make big amount of processes. And then Load Average goes to 200+. Have 128gb RAM and 32 cores. After that mane users have 502 error. It helps reboot php-fpm, but not for long, the processes multiply again and the load increases.

Tried different php max child values.

php.ini :
engine = On
short_open_tag = On
asp_tags = Off
precision = 14
output_buffering = Off
zlib.output_compression = Off
implicit_flush = Off
serialize_precision = 17
expose_php = Off
max_execution_time = 1000
max_input_time = 0
max_input_vars = 10000
error_reporting = E_ERROR
error_reporting = 
display_errors = Off
display_startup_errors = On
log_errors = On
log_errors_max_len = 1024
ignore_repeated_errors = On
ignore_repeated_source = Off
report_memleaks = On
track_errors = On
html_errors = On
error_log = /var/log/php-fpm/php-errors.log
variables_order = "GPCS"
request_order = "GP"
register_argc_argv = Off
auto_globals_jit = On
post_max_size = 100M
default_mimetype = "text/html"
default_charset = "UTF-8"
enable_dl = Off
file_uploads = On
upload_tmp_dir =/tmp/php/uploads
upload_max_filesize = 200M
max_file_uploads = 50
allow_url_fopen = On
allow_url_include = Off
default_socket_timeout = 60000
extension=memcache.so
extension=memcached.so
zend_extension=ioncube_loader_lin_7.1.so
cli_server.color = On
date.timezone = "Europe/Kiev"
SMTP = localhost
smtp_port = 25
sendmail_path="/usr/sbin/sendmail -t -i"
mail.add_x_header = On
mail.log =/var/log/php-fpm/php-mail.log
sql.safe_mode = Off
mysql.allow_local_infile = On
mysql.allow_persistent = On
mysql.cache_size = 2000
mysql.max_persistent = -1
mysql.max_links = -1
mysql.default_port =
mysql.default_socket =
mysql.default_host =
mysql.default_user =
mysql.default_password =
mysql.connect_timeout = 3
mysql.trace_mode = Off
mysqli.max_persistent = -1
mysqli.allow_persistent = On
mysqli.max_links = -1
mysqli.cache_size = 2000
mysqli.default_port = 3306
mysqli.default_socket =
mysqli.default_host =
mysqli.default_user =
mysqli.default_pw =
mysqli.reconnect = Off
mysqlnd.collect_statistics = On
mysqlnd.collect_memory_statistics = On
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0
session.save_handler = files
session.save_path = "/tmp/php/sessions"
session.use_cookies = 1
;session.cookie_secure =
session.use_only_cookies = 1
session.name = PHPSESSID
session.auto_start = 0
session.cookie_lifetime = 0
session.cookie_path = /
session.cookie_domain =
session.cookie_httponly =
session.serialize_handler = php
session.gc_probability = 1
session.gc_divisor = 1000
session.gc_maxlifetime = 1440
session.bug_compat_42 = On
session.bug_compat_warn = On
session.referer_check =
session.cache_limiter = nocache
session.cache_expire = 180
session.use_trans_sid = 0
session.hash_function = 0
session.hash_bits_per_character = 5
tidy.clean_output = Off
soap.wsdl_cache_enabled=1
soap.wsdl_cache_dir="/tmp"
soap.wsdl_cache_ttl=86400
soap.wsdl_cache_limit = 5
ldap.max_links = -1

ngix.conf:
worker_processes 32; 
worker_rlimit_nofile 200000; 
load_module "modules/ngx_http_geoip_module.so";

events {
    worker_connections  8000;  
    use epoll;
    multi_accept on;  
}

http {
    ssl_session_cache   shared:SSL:100m;
    ssl_session_timeout 1h;
    include       mime.types;
    default_type  application/octet-stream;
    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites/*;
    server_tokens off;
    sendfile            on;
    keepalive_timeout   65;
    keepalive_requests 1000;  ###
    reset_timedout_connection on;  ###
    open_file_cache max=200000 inactive=20s;  #####
    open_file_cache_valid 30s; ####
    open_file_cache_min_uses 2; ###
    open_file_cache_errors on; ###
    client_body_timeout 10;  ###
    send_timeout 2;  ###
    chunked_transfer_encoding off;
    server_names_hash_max_size 1024;
    server_names_hash_bucket_size 256;
    client_max_body_size 128m;
    proxy_send_timeout 6000;
    proxy_read_timeout 600;
    large_client_header_buffers 16 32k;
    tcp_nodelay on;
    tcp_nopush on;   ####
    include cloudflare;

    #Compression.
    gzip                on;
    gzip_disable "msie6";
    gzip_vary on;
    gzip_proxied any;
    gzip_comp_level 8;
    gzip_buffers 16 8k;
    gzip_http_version 1.1;
    gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;


}

fpm.conf:
listen = 127.0.0.1:9001
user = website
group = website
request_slowlog_timeout = 3s
listen.allowed_clients = 127.0.0.1
pm = ondemand
pm.max_children = 4000
pm.start_servers = 5
pm.min_spare_servers = 5
pm.process_idle_timeout = 1s
pm.max_requests = 0
listen.backlog = 8193
pm.status_path = /php-status.html
request_terminate_timeout = 90s
rlimit_files = 131072
rlimit_core = unlimited
catch_workers_output = yes
env[HOSTNAME] = $HOSTNAME
env[TMP] = /tmp
env[TMPDIR] = /tmp
env[TEMP] = /tmp
catch_workers_output = yes
php_flag[display_errors] = on
php_admin_value[error_log] = /var/log/fpm-php.www.log
php_admin_flag[log_errors] = on

Can someone help with conf files? Maby something is wrong. He was made for a long time ago and not by me.

Answer 1

Your problem is not about PHP config at this point, IMHO, your problem is about scalability.

You didn't mentionned if your server is statefull or stateless, I suppose, due to your problem that it's statefull. Plus, you didn't explained if you have a cluster of servers with load balancing, but I assume you've not.

What does that mean ?

It means that your server, for each user, write a file to store its session in the server filesystem. This process is memory consumer, and its expensive.

A simple math calcul will show your problem,

you said that you have 128G of RAM, and about 3000 users, means they consume all the ram available if your PHP memory limit is about 32M per user. Wich is basically extremely low to be honest in 2019.

Solutions ?

For 3000 user at the same time on a single server, you should consider load balancing. Load balancing allow your app to be balanced between multiple servers, depends on the trafic. It involves a lot of setup and configurations to start with, but it's doable.

Another solution is to migrate your application to a stateless one. Means that the server does not know about users session, and authentication is done by a token (JWT to the rescue). This solution is not easy to settle, if your app is big, and require some work also.

A link to know more (look for the Performance and Scalability with PHP Sessions section)