524 time out error with mutiiple connections

Question

I am testing server performance using JMeter to mimic a large number of users hitting our digital ocean server in a short period of time. When I set JMeter to 200 users and test it against my Laravel based webpage, every works fine. When I increase the number of users to 500, I start to get 524 errors.

The server CPU never goes over 10% and the memory is at 30%, so the server has enough power. The first couple of hundred requests process correctly, but then the 524 errors begin to appear. The failed requests have a higher latency and connection than the successful requests as shown in this screen shot. Any clue where I should start looking for the problem?

My conf file in sites available

location ~.php$ {

try_files $uri =404;

fastcgi_pass unix:/var/run/php-fpm/xxxxxxxxxxxx.com.sock;
fastcgi_index index.php;

}

my nginx.conf settings

fastcgi_buffers 8 128k;
  fastcgi_buffer_size 256k;

  client_header_timeout 3000;
  client_body_timeout 3000;
  fastcgi_read_timeout 3000;
  client_max_body_size 32m;

my /etc/sysctl.conf file settings, taken from this post after previously getting 504 errors https://www.digitalocean.com/community/questions/getting-nginx-fpm-sock-error

# sysctl settings are defined through files in
# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
#
# Vendors settings live in /usr/lib/sysctl.d/.
# To override a whole file, create a new file with the same in
# /etc/sysctl.d/ and put new settings there. To override
# only specific settings, add a file with a lexically later
# name in /etc/sysctl.d/ and put new settings there.
#
# For more information, see sysctl.conf(5) and sysctl.d(5).

### IMPROVE SYSTEM MEMORY MANAGEMENT ###

# Increase size of file handles and inode cache
fs.file-max = 2097152

# Do less swapping
vm.swappiness = 10
vm.dirty_ratio = 60
vm.dirty_background_ratio = 2

### GENERAL NETWORK SECURITY OPTIONS ###

# Number of times SYNACKs for passive TCP connection.
net.ipv4.tcp_synack_retries = 2

# Allowed local port range
net.ipv4.ip_local_port_range = 2000 65535

# Protect Against TCP Time-Wait
net.ipv4.tcp_rfc1337 = 1

# Decrease the time default value for tcp_fin_timeout connection
net.ipv4.tcp_fin_timeout = 15

# Decrease the time default value for connections to keep alive
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_intvl = 15

### TUNING NETWORK PERFORMANCE ###

# Default Socket Receive Buffer
net.core.rmem_default = 31457280

# Maximum Socket Receive Buffer
net.core.rmem_max = 12582912

# Default Socket Send Buffer
net.core.wmem_default = 31457280

# Maximum Socket Send Buffer
net.core.wmem_max = 12582912

# Increase number of incoming connections
net.core.somaxconn = 65535

# Increase number of incoming connections backlog
net.core.netdev_max_backlog = 65535

# Increase the maximum amount of option memory buffers
net.core.optmem_max = 25165824

# Increase the maximum total buffer-space allocatable
# This is measured in units of pages (4096 bytes)
net.ipv4.tcp_mem = 65535 131072 262144
net.ipv4.udp_mem = 65535 131072 262144

# Increase the read-buffer space allocatable
net.ipv4.tcp_rmem = 8192 87380 16777216
net.ipv4.udp_rmem_min = 16384

# Increase the write-buffer-space allocatable
net.ipv4.tcp_wmem = 8192 65535 16777216
net.ipv4.udp_wmem_min = 16384

# Increase the tcp-time-wait buckets pool size to prevent simple DOS attacks
net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1

Added this to /etc/security/limits.conf

nginx           soft    nofile          2097152
nginx           hard    nofile          2097152
www-data        soft    nofile          2097152
www-data        hard    nofile          2097152

My php-fpm.d conf file settings

pm = static
pm.max_children = 40
pm.start_servers = 8
pm.min_spare_servers = 4
pm.max_spare_servers = 8
; number of processes to process before respawning. Lower number if you have memory leaks, but each respawn takes time
pm.max_requests=50
; pm.process_idle_timeout=10

chdir = /

php_admin_value[disable_functions] = exec,passthru,shell_exec,system
php_admin_flag[allow_url_fopen] = on
php_admin_flag[log_errors] = on

php_admin_value[post_max_size] = 8M
php_admin_value[upload_max_filesize] = 8M

Answer 1

HTTP 524 errors are cloudflare specific - they're not being generated by your own nginx installation. Cloudflare gave up waiting for a response from your backend service, probably because of the low numbers of fpm children available to serve the requests.

524 A Timeout Occurred

Cloudflare was able to complete a TCP connection to the origin server, but did not receive a timely HTTP response.

If you're performance testing your own server setup, don't go through the endpoint that points to cloudflare.

The general "backend timed out" response for http servers are 504 Gateway Timeout.