CODEWITHSUNDEEP
pythonweb.pygevent
Pradeep Banavara
Pradeep Banavara

Reputation: 1003

Gevent joinall blocking error

A bit of background on the application:

  1. Clients make web requests
  2. Server has to handle web requests ( each request takes 20 seconds )
  3. Send response to clients.

My approach was to first parallelize the request serving part using webpy and mod_wsgi. If I start the code with 20 or so threads, but since each thread takes 20 seconds to complete, the request level multi-threading is of less use. So I had to reduce the 20 seconds which I did by spawning greenlets. Something like this

File wordprocess_gevent.py

import gevent
urls = ('/'. A)
class A:
  output = {}
  def POST(self):
    #words is a list of words
    method A(words):

  def A(self, words):
    threads = []
    for word in words:
      i_thread = gevent.spawn(B, word)
      threads.append(i_thread)
    gevent.joinall(threads, timeout=2)

  def B(self, word):
    #Process word takes 20 seconds
    result = process(word)
    a[word] = result

application = web.application(urls, globals()).wsgifunc()

I start this code using the mod_wsgi-express as given below:

mod_wsgi-express start-server wordprocess_gevent.py --processes 5 --server-root wsgi_logs/ --with-wdb &

When simultaneous POST requests arrive I get this error

LoopExit: This operation would block forever

at the line 

gevent.joinall(threads, timeout=2)

BUT if I post a single POST request - I get the required results. Can someone help me out here please.

Upvotes: 1

Views: 586

Answers (1)

Pradeep Banavara
Pradeep Banavara

Reputation: 1003

So I solved this by removing mod_wsgi altogether. Even with one process I was getting the same result when multiple requests came.

Added the following and its now working like a charm :)

if __name__ == "__main__":
  application = web.application(urls, globals()).wsgifunc()
  appserver = WSGIServer(('', 8000), application)
  appserver.serve_forever()

Upvotes: 1

Related Questions