CODEWITHSUNDEEP
tornadononblocking
Gaurav
Gaurav

Reputation: 627

Tornado thread-safe non-blocking operation

I got a conceptual doubt about Tornado.

I have a list of queries which I am running against the twitter api.

So I have something like

while (keywords):

if (requests.remaining > 300) :
self.twitter_request(path= "/search",access_token=self.current_user["access_token"],
        callback=(yield gen.Callback("key")),q=word)
response = yield gen.Wait("key")
self.parse_response(response)

else:
  sleep(3000)

And parse_response is defined as 

def parse_response(self,response):
  //parse the response object
  //check if number of results returned is 100. If yes then, check current page number.
  //if less than 4 then
     self.twitter_request(path= "/search",access_token=self.current_user["access_token"],
        callback=self.async_callback(self._parse_response,fileIter= fileIter,fileType= fileType, query= query),q=query,rpp="100")

return

So basically my parse_response function is parsing the response and requesting more pages for the same keyword till page_number is 4.

But in the main loop which calls this auxiliary function, gets suspended when I call time.sleep(3000). This would even make the background functions to sleep since the main thread is sleeping and they would resume once this main thread is alive. (Since I made requests for the keywords initially and the parse_Response is making use of async_Callback, these async callbacks are not executed when sleep(3000) command gets executed). Is there a way to allow the background processed to run even when the main class issues a sleep command.

Upvotes: 0

Views: 2176

Answers (1)

Nikolay Fominyh
Nikolay Fominyh

Reputation: 9256

Open for yourself tornado periodic callback. And you will get something like this. 

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import tornado.web
import tornado.httpserver
import tornado.ioloop

KEYWORDS = []

class MyApplication(tornado.web.Application):

  def period_run(self):
    while(KEYWORDS):
        if (requests.remaining > 300) :
            self.twitter_request(path="/search",access_token=self.current_user["access_token"],
            callback=(yield gen.Callback("key")),q=word)
            response = yield gen.Wait("key")
            self.parse_response(response)

app = MyApplication(handlers=urls, **settings)

http_server = tornado.httpserver.HTTPServer(app, xheaders=True)

loop = tornado.ioloop.IOLoop.instance()
TIME_TO_RUN = 400
period_cbk = tornado.ioloop.PeriodicCallback(app.period_run, 3000 + TIME_TO_RUN, loop)
period_cbk.start()
loop.start()

More info can be found here.

Upvotes: 1

Related Questions