CODEWITHSUNDEEP
ruby-on-railsamazon-web-servicesnginxpassenger
Vitali Grabovski
Vitali Grabovski

Reputation: 183

How to configure Nginx with Passenger to process 500 requests/sec

I am hosting ruby application but it gives errors (some kind of error 500) while processing more than 400 requests per second. Test (loader.io) with lower amount of requests (below 400 request) finishes with good results. I think I could have better results while processing 500 requests/second and more.

The app is using t2.2xlarge ec2 instance (with 32Gb of memory, 8 virtual cores). I guess it might give better performance. The machine is running on Ubuntu 14.04, Rails 4.0.12, Nginx with Passenger.

I tried to make some changes in Nginx configuration but without any big progress. My current configuration:

passenger_max_pool_size 60;
#passenger_pool_idle_time 20;
server {
  listen 80;
  return 301 https://mydomain.eu$request_uri;
}

server {
  listen 443;
  server_name ~^(\w+)\.mydomain.eu$;
  return 301 https://mydomain.eu$request_uri;
}
server {
  listen 443 ssl spdy default;
  server_name mydomain.eu;
  passenger_enabled on;
  #passenger_max_pool_size 12;
  passenger_max_request_queue_size 2000;
  gzip on;

  root /home/ubuntu/application/cversion/public;

  ssl                  on;
  ssl_certificate      /home/ubuntu/fvhsdvhfd35/ssl-bundle1.crt;
  ssl_certificate_key  /home/ubuntu/fvhsdvhfd35/prvt.key;
  ssl_session_timeout  5m;
  ssl_protocols        TLSv1 TLSv1.1 TLSv1.2 SSLv3;
  ssl_ciphers          "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS";
  ssl_prefer_server_ciphers  on;

  location = /favicon.png {
    expires    max;
    add_header Cache-Control public;
  }

  location = /ZeroClipboard.swf {
    expires    max;
    add_header Cache-Control public;
  }

  location ~ ^/(assets)/  {
    gzip_static on;
    expires     max;
    add_header  Cache-Control public;
  }

  # disable gzip on all omniauth paths to prevent BREACH
  location ~ ^/auth/ {
    gzip off;
    passenger_enabled on;
  }

Do you have any idea how to get more than 400 request per second?

here is a log of Nginx with Passenger while processing 500 requests(with passenger_max_pool_size 30; and passenger_max_request_queue_size 1200;)

2017/07/06 01:58:32 [error] 11749#11749: *56391 connect() to unix:/tmp/passenger.e1PiPXp/agents.s/core failed (11: Resource temporarily unavailable) while connecting to upstream, client: 54.89.44.6, server: mydomain.us, request: "GET / HTTP/1.1", upstream: "passenger:unix:/tmp/passenger.e1PiPXp/agents.s/core:", host: "mydomain.us"
2017/07/06 01:58:32 [error] 11740#11740: *64104 connect() to unix:/tmp/passenger.e1PiPXp/agents.s/core failed (11: Resource temporarily unavailable) while connecting to upstream, client: 52.87.219.148, server: mydomain.us, request: "GET / HTTP/1.1", upstream: "passenger:unix:/tmp/passenger.e1PiPXp/agents.s/core:", host: "mydomain.us"
2017/07/06 01:58:32 [error] 11749#11749: *64251 connect() to unix:/tmp/passenger.e1PiPXp/agents.s/core failed (11: Resource temporarily unavailable) while connecting to upstream, client: 52.87.219.148, server: mydomain.us, request: "GET / HTTP/1.1", upstream: "passenger:unix:/tmp/passenger.e1PiPXp/agents.s/core:", host: "mydomain.us"
2017/07/06 01:58:32 [error] 11749#11749: *63289 connect() to unix:/tmp/passenger.e1PiPXp/agents.s/core failed (11: Resource temporarily unavailable) while connecting to upstream, client: 54.89.44.6, server: mydomain.us, request: "GET / HTTP/1.1", upstream: "passenger:unix:/tmp/passenger.e1PiPXp/agents.s/core:", host: "mydomain.us"
2017/07/06 01:58:32 [error] 11748#11748: *67786 connect() to unix:/tmp/passenger.e1PiPXp/agents.s/core failed (11: Resource temporarily unavailable) while connecting to upstream, client: 52.86.198.91, server: mydomain.us, request: "GET / HTTP/1.1", upstream: "passenger:unix:/tmp/passenger.e1PiPXp/agents.s/core:", host: "mydomain.us"
2017/07/06 01:58:32 [error] 11748#11748: *35057 connect() to unix:/tmp/passenger.e1PiPXp/agents.s/core failed (11: Resource temporarily unavailable) while connecting to upstream, client: 52.86.198.91, server: mydomain.us, request: "GET / HTTP/1.1", upstream: "passenger:unix:/tmp/passenger.e1PiPXp/agents.s/core:", host: "mydomain.us"
2017/07/06 01:58:32 [error] 11748#11748: *35166 connect() to unix:/tmp/passenger.e1PiPXp/agents.s/core failed (11: Resource temporarily unavailable) while connecting to upstream, client: 52.86.198.91, server: mydomain.us, request: "GET / HTTP/1.1", upstream: "passenger:unix:/tmp/passenger.e1PiPXp/agents.s/core:", host: "mydomain.us"
2017/07/06 01:58:32 [error] 11744#11744: *43208 connect() to unix:/tmp/passenger.e1PiPXp/agents.s/core failed (11: Resource temporarily unavailable) while connecting to upstream, client: 52.86.198.91, server: mydomain.us, request: "GET / HTTP/1.1", upstream: "passenger:unix:/tmp/passenger.e1PiPXp/agents.s/core:", host: "mydomain.us"
2017/07/06 01:58:32 [error] 11744#11744: *69130 connect() to unix:/tmp/passenger.e1PiPXp/agents.s/core failed (11: Resource temporarily unavailable) while connecting to upstream, client: 54.162.105.71, server: mydomain.us, request: "GET / HTTP/1.1", upstream: "passenger:unix:/tmp/passenger.e1PiPXp/agents.s/core:", host: "mydomain.us"
[ 2017-07-06 01:58:34.3865 11703/7fc703fff700 Ser/AcceptLoadBalancer.h:150 ]: Resuming accepting new clients

UPDATE

I got a solution. These changes of Nginx Configuration gave me 1000 request/second performance.

At first I put: 

"65536" in /proc/sys/net/core/somaxconn
"65536" in /proc/sys/net/ipv4/tcp_max_syn_backlog

/etc/nginx/conf.d/m.conf:

    passenger_max_pool_size 90;
    passenger_socket_backlog 16384;

    #in server block
    #was listen 443 ssl spdy default;
    listen 443 ssl spdy default backlog=16384;
    passenger_max_request_queue_size 2300;
    ssl_session_cache shared:SSL:10m;

/etc/nginx/nginx.conf:

worker_rlimit_nofile 131072;

#in events block:
use epoll;
worker_connections 8192;

Another question

Average response time is about 6seconds for 1000 requests per second during 1 minute of tests. Any ideas how to improve average response time for this amount of requests?

UPDATE2

I change my Nginx config to enable Nginx Microcashing according to this blog, but I've got no better performance. 500 req/second gave me 5.1sec as average response time. About 900 req/sec - 5.5sec. However without caching I have 2.5sec for 500 requests and 5.6 sec for 900 requests.

/etc/nginx/nginx.conf:

    ...
    http {
      ...
      proxy_cache_path /tmp/cache keys_zone=one:10m levels=1:2 inactive=600s max_size=100m;
      ...
    }

/etc/nginx/conf.d/m.conf:

}

passenger_max_pool_size 90;
#passenger_pool_idle_time 20;
passenger_socket_backlog 16384;
server {
  listen 80;
  return 301 https://mydomain.eu$request_uri;
}

server {
  listen 443;
  server_name ~^(\w+)\.mydomain.eu$;
  return 301 https://mydomain.eu$request_uri;
}
server {
  listen 443 ssl spdy default backlog=16384;
  server_name mydomain.eu;

  ssl                  on;
  ssl_certificate      /home/ubuntu/fvhsdvhfd35/ssl-bundle1.crt;
  ssl_certificate_key  /home/ubuntu/fvhsdvhfd35/prvt.key;
  ssl_session_timeout  5m;
  ssl_protocols        TLSv1 TLSv1.1 TLSv1.2 SSLv3;
  ssl_ciphers          "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS";
  ssl_prefer_server_ciphers  on;

  ssl_session_cache shared:SSL:10m;

  location / {

    proxy_http_version 1.1; # Always upgrade to HTTP/1.1
    proxy_set_header Connection ""; # Enable keepalives
    proxy_set_header Accept-Encoding ""; # Optimize encoding
    proxy_pass http://127.0.0.1:81/;

    proxy_cache one;
    proxy_cache_lock on;
    proxy_cache_valid 200 1s;
    proxy_cache_use_stale updating;
  }
}
server {

  listen 81;
  server_name mydomain.eu;
  passenger_enabled on;

  passenger_max_request_queue_size 2300;
  gzip on;

  root /home/ubuntu/application/cversion/public;


  location = /favicon.png {
    expires    max;
    add_header Cache-Control public;
  }

  location = /ZeroClipboard.swf {
    expires    max;
    add_header Cache-Control public;
  }

  location ~ ^/(assets)/  {
    gzip_static on;
    expires     max;
    add_header  Cache-Control public;
  }

  # disable gzip on all omniauth paths to prevent BREACH
  location ~ ^/auth/ {
    gzip off;
    passenger_enabled on;
  }

}

Upvotes: 3

Views: 3922

Answers (2)

Akshay Chhikara
Akshay Chhikara

Reputation: 56

For making these optimizations make sure you refer the link nginx blog and do consider the response time for each request (minimise it as much as you can using rails techniques). Also consider your database optimizations i.e using the right indexes and maximum number of database connections concurrently. As it is a multi level problem and configurations must be made at each level for best performance. Good luck :)

Upvotes: 2

Tushar Pal
Tushar Pal

Reputation: 523

Nginx configuration can be reviewed according to official performance tuning recommendation: https://www.nginx.com/blog/tuning-nginx/

Upvotes: 1

Related Questions