CODEWITHSUNDEEP
pythonmultithreadinguwsgigevent
ctpaquette
ctpaquette

Reputation: 118

Running Out of Threads: UWSGI + Multithreaded Python Application with GeventHTTPClient

I'm currently running a python web API that is NOT multithreaded with much success on the uWSGI + NGINX stack. Due to new operational needs, I have implemented a new build that includes multithreaded requests to external data sources. However, when I deploy this new multithreaded build under uWSGI with --enable-threads, after a few minutes, the machine runs out of available threads.

I was able to isolate the issue to my usage of geventhttpclient for my external HTTP requests by monitoring the thread count using ps -eLf | grep <process id>| wc -l. I have currently 2 worker threads (two external requests) in my application, so as I noticed, every time I hit/make a request from my API, the application thread use count increases by 2. If I swap my use of geventhttpclient with the standard python Requests module in just one of these worker threads, the thread count only increases by 1.

NOTE: I am using HTTPClient.close() to close the connection within each thread.

This leads me to suspect that geventhttpclient creates new threads that do not terminate when used in multithreaded uWSGI applications.

Is there an easy way around this chokepoint? The performance of geventhttpclient is exceptional in non-multithreaded uWSGI applications, so I would love to continue using this.

Thanks and let me know if I can provide any more information.

Upvotes: 1

Views: 898

Answers (1)

roberto
roberto

Reputation: 12933

Mixing non-blocking programming (geventhttpclient) with blocking one (a uWSGI thread/process) is completely wrong. This is a general rule: even if your app is 99% non blocking it is still blocking. This is amplified by the fact that gevent makes use of stack switching to simulate blocking programming paradigms.

This is like cooperative multitasking, and it is managed by the so called 'gevent-hub'. Unfortunately, albeit your greenlets will be able to make http requests they will be never terminated because the gevent hub will never run again once the request is over.

If you want to maintain the geventhttpclient approach you have to set uWSGI in gevent mode, but you need to be sure that all the modules and techniques used by your app are gevent friendly.

Upvotes: 1

Related Questions