CODEWITHSUNDEEP
nginxflaskuwsgi
Aaron Wojnowski
Aaron Wojnowski

Reputation: 6480

uWSGI workers all busy, yet requests per second is very low

I'm having quite a bit of trouble optimizing uWSGI to serve my Flask application, which receives around 1500 requests per second.

I currently am running Nginx on my front-facing server with four servers (each an AWS m5-xlarge, so there's plenty of computing power) in an upstream block. The front-facing server connects to the uWSGI servers on port 5000 using web sockets. I have verified that this is all working and that Nginx is capable of handling the load.

Here is my uWSGI configuration:

[uwsgi]
module = wsgi:app

master = true

processes = 24
threads = 4
enable-threads = True

socket = 0.0.0.0:5000
stats = api.stats.sock
chmod-socket = 660
vacuum = true

die-on-term = true

Here is a screenshot of the typical uwsgitop stats for each server:

uWSGI Stats

As you can see, the requests per second is extremely low yet the workers are still busy.

The top command shows that my server utilization is extremely low.

top

Meanwhile, Nginx is spitting out thousands of errors such as:

*5151324 upstream timed out (110: Connection timed out) while connecting to upstream

If anyone can help me diagnose this, that would be super appreciated.

Upvotes: 4

Views: 3373

Answers (2)

Tuesleep
Tuesleep

Reputation: 119

uWSGI all worker busy, possible the queue is blocking. Check database or other operation which cost too much time.

I had similar issue like you, and solved is Postgresql 12 problem, the connections up to limit. Default max_connections is 100. I changed this config to 10000.

alter system set max_connections='10000';

and restart db

systemctl restart postgresql-12

That's solved, maybe it's a reference, maybe not similar reason, I hope it's helpful.

Upvotes: -1

wonton
wonton

Reputation: 8247

I had a similar issue where my listen queue was growing and the RPS was low despite all the workers idling.

There are a lot of potential causes

  • net.ipv4.ip_conntrack_max not high enough
  • net.core.somaxconn not high enough
  • ulimit not high enough
  • uwsgi --listen not high enough
  • nginx worker processes too low
  • nginx worker connections too low

If none of these work, then you'll want to check your logs that inbound requests to uWSGI are under http/1.1 and not http/1.0, and then use --http11-socket

Here are some of my findings when I wrestled with this issue: https://wontonst.blogspot.com/2019/06/squishing-performance-bug-in.html

Upvotes: 2

Related Questions