CODEWITHSUNDEEP
node.jsnginxconcurrencyubuntu-14.04
Anthony Chung
Anthony Chung

Reputation: 1477

10K concurrent connections with nginx

I'm building an app on my own and I'd like for it to handle 10K concurrent connections (tested via local machine and locust script)

Hosted on two ubuntu 14.04 servers with nginx reverse proxy and nodeJS app server.

Currently I get to around 3.3K concurrent users before suffering a spike of 500 connection dropped errors.

I achieved load balancing across port connections by running the app on two separate ports and using an upstream directive to spread requests over ports.

However, this has not shown any demonstrated improvement in my numbers.

Question:

  1. I know that there is a lot of information missing here (I imagine how much bandwidth each user requires). How do I go about gathering the right information to decide this?

  2. What other option can I consider/learn/implement to generate the biggest gain in possible concurrent users?

Thanks a lot!

Upvotes: 3

Views: 6701

Answers (1)

Peter Perron
Peter Perron

Reputation: 326

If you see a lot of "Too many open files" errors in your nginx log, it is because you've reach the limit of concurrent open sockets currently set on the server.

You might have to increase the Usage Limit (ULimit) for the nginx user and probably the node app user too. This is the first issue I hit each time I load test an nginx+nodejs. (https://www.cyberciti.biz/faq/linux-unix-nginx-too-many-open-files/)

ULimit

The ulimit command gives you the limits of the current user so you have to run it while connected on the nginx user

# su in the user nginx
su - nginx
ulimit -Hn
ulimit -Sn

# or
su nginx --shell /bin/bash --command "ulimit -Hn"
su nginx --shell /bin/bash --command "ulimit -Sn"

We usually change it in /etc/security/limits.conf

sudo vi /etc/security/limits.conf

# Add the following lines
nginx       soft    nofile   30000
nginx       hard    nofile  30000
# Save

# Reload (I think rebooting here is the best way)
sysctl -p

nginx.conf

After that, you have to set the limit at the nginx software level in the nginx config file

sudo vi /etc/nginx/nginx.conf

# Add this line at the root of the config
worker_rlimit_nofile 30000;

# Reload (This will fail if you have SELinux enabled)
sudo nginx -s reload

SELinux

To allow the nginx app to set its own limit you can set a selinux bool:

setsebool httpd_setrlimit on
sudo nginx -s reload

Upvotes: 4

Related Questions