CODEWITHSUNDEEP
pythonflaskuwsgi
licorna
licorna

Reputation: 5870

uWSGI, Flask and Parallel execution blocks

I have a small API written in Python with Flask. It is a wrapper API around another API (which I don't control). I need to make several requests and I want to make them parallel. The first alternative was to use grequests but inexplicably the uwsgi server hanged and didn't respond. It took a while for it to reload workers after sending HUP signal to the master process. The code I was using is something like:

urls = [
    'http://www.example.com', 
    'http://www.example.net',
]
urls = [grequests.get(x, hooks=dict(response=function_to_do_some_work))
        for url in urls]
grequests.map(urls) # <- here it hangs

I thought it was some problem with gevent/grequests so I reimplemented the solution using multiprocessing.Pool, like this:

urls = [
    'http://www.example.com', 
    'http://www.example.net',
]
pool = multiprocessing.Pool(10)
pool.map(function_to_do_some_work, urls) # <- here it hangs

The problem here is not the code, that runs perfectly in the development environment. The problem is with the uwsgi and Python execution model, or something between those lines. I don't know where to start looking for answers here. The versions of the modules I'm using are:

Python 2.7.5
grequests 0.2.0
Flask 0.10.1
uwsgi 2.0.9

What alternative of parallel execution should I use to perform several tasks at the same time, under this execution environment?

Upvotes: 2

Views: 1687

Answers (1)

tourdownunder
tourdownunder

Reputation: 1827

Threading needs to be enabled on uwsgi by 

enable-threads = True

Full example of config of nginx, and uwsgi, upstart, virtual-env see

https://stackoverflow.com/a/27221427/567606

Upvotes: 3

Related Questions