CODEWITHSUNDEEP
pythonredistornado
Tampa
Tampa

Reputation: 78254

Python tornado redis - how to do a pipeline

What is the proper way do to a pipeline in tornadoredis? The documentation is not clear. I am using the tornadoredis libary. 

    import tornadoredis

    @tornado.web.asynchronous
    @tornado.gen.engine
    def get(self):
        self.pipe_threads = []
        sw = stopwatch.Timer()
        if not self.get_cookie('rtbhui'):
            self.cookie_id=str(uuid4())
            self.set_cookie('test', self.cookie_id, domain='rtb.rtbhui.com',expires_days=365*2)
        else:
            self.cookie_id = self.get_cookie('test')

        t = yield tornado.gen.Task(c.pipeline)
        yield tornado.gen.Task(t.set,'key', 'value')
        yield tornado.gen.Task(t.execute)



starting
ERROR:root:Uncaught exception GET /apixel?bid=eydhdmlkJzogJ2luZm9AYWRtYWdpYy5jby5qcCcsICdjcmlkJzogJzIwNzY3MDczNTE1JywgJ21hYmlkJzogeydub2RlJzogTm9uZSwgJ2hvZCc6ICcxMycsICdjaWQnOiAnMzYnLCAnaW5kdXN0cnknOiBOb25lLCAnZXgnOiAnMScsICd2aWQxJzogJzI5JywgJ2Rvdyc6ICczJ30sICdwcmlkJzogeydob2QnOiAnMTMnLCAnd295JzogJzE4JywgJ2NpZCc6ICczNicsICdkb3cnOiAnMycsICdzc3AnOiAnYWR4JywgJ3N0JzogTm9uZSwgJ2J0JzogJ2ZpcmVmb3gnLCAnY3R5JzogJ3Rva3lvJywgJ2V4JzogJzEnLCAndmlkMic6ICcyMjInLCAnZHQnOiAnMScsICdvcyc6ICdtYWMnLCAndmlkMSc6ICcyOSd9LCAnYWdpZCc6ICc0NTQ3OTE3Nzk1JywgJ2Nvb2tpZWlkJzogJ3JldGFnZXRpbmc6Y29va2llJywgJ2RpZCc6ICd5YWhvby5jb20nLCAndmFsaWRhdGlvbic6IFRydWV9 (127.0.0.1)
HTTPRequest(protocol='http', host='127.0.0.1:8055', method='GET', uri='/apixel?bid=eydhdmlkJzogJ2luZm9AYWRtYWdpYy5jby5qcCcsICdjcmlkJzogJzIwNzY3MDczNTE1JywgJ21hYmlkJzogeydub2RlJzogTm9uZSwgJ2hvZCc6ICcxMycsICdjaWQnOiAnMzYnLCAnaW5kdXN0cnknOiBOb25lLCAnZXgnOiAnMScsICd2aWQxJzogJzI5JywgJ2Rvdyc6ICczJ30sICdwcmlkJzogeydob2QnOiAnMTMnLCAnd295JzogJzE4JywgJ2NpZCc6ICczNicsICdkb3cnOiAnMycsICdzc3AnOiAnYWR4JywgJ3N0JzogTm9uZSwgJ2J0JzogJ2ZpcmVmb3gnLCAnY3R5JzogJ3Rva3lvJywgJ2V4JzogJzEnLCAndmlkMic6ICcyMjInLCAnZHQnOiAnMScsICdvcyc6ICdtYWMnLCAndmlkMSc6ICcyOSd9LCAnYWdpZCc6ICc0NTQ3OTE3Nzk1JywgJ2Nvb2tpZWlkJzogJ3JldGFnZXRpbmc6Y29va2llJywgJ2RpZCc6ICd5YWhvby5jb20nLCAndmFsaWRhdGlvbic6IFRydWV9', version='HTTP/1.1', remote_ip='127.0.0.1', body='', headers={'Connection': 'keep-alive', 'Accept-Language': 'en-us,en;q=0.5', 'Accept-Encoding': 'gzip, deflate', 'Cache-Control': 'max-age=0', 'Host': '127.0.0.1:8055', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:11.0) Gecko/20100101 Firefox/11.0'})
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/tornado-2.2.1-py2.7.egg/tornado/web.py", line 1074, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tornado-2.2.1-py2.7.egg/tornado/gen.py", line 107, in wrapper
    runner.run()
  File "/usr/local/lib/python2.7/dist-packages/tornado-2.2.1-py2.7.egg/tornado/gen.py", line 319, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/ubuntu/workspace/rtbopsConfig/rtbServers/rtbTornadoServer/tornadoServer.py", line 81, in get
    t = yield tornado.gen.Task(c.pipeline)
  File "/usr/local/lib/python2.7/dist-packages/tornado-2.2.1-py2.7.egg/tornado/gen.py", line 341, in run
    self.yield_point.start(self)
  File "/usr/local/lib/python2.7/dist-packages/tornado-2.2.1-py2.7.egg/tornado/gen.py", line 223, in start
    self.func(*self.args, **self.kwargs)
TypeError: pipeline() got an unexpected keyword argument 'callback'
ERROR:root:500 GET /apixel?bid=eydhdmlkJzogJ2luZm9AYWRtYWdpYy5jby5qcCcsICdjcmlkJzogJzIwNzY3MDczNTE1JywgJ21hYmlkJzogeydub2RlJzogTm9uZSwgJ2hvZCc6ICcxMycsICdjaWQnOiAnMzYnLCAnaW5kdXN0cnknOiBOb25lLCAnZXgnOiAnMScsICd2aWQxJzogJzI5JywgJ2Rvdyc6ICczJ30sICdwcmlkJzogeydob2QnOiAnMTMnLCAnd295JzogJzE4JywgJ2NpZCc6ICczNicsICdkb3cnOiAnMycsICdzc3AnOiAnYWR4JywgJ3N0JzogTm9uZSwgJ2J0JzogJ2ZpcmVmb3gnLCAnY3R5JzogJ3Rva3lvJywgJ2V4JzogJzEnLCAndmlkMic6ICcyMjInLCAnZHQnOiAnMScsICdvcyc6ICdtYWMnLCAndmlkMSc6ICcyOSd9LCAnYWdpZCc6ICc0NTQ3OTE3Nzk1JywgJ2Nvb2tpZWlkJzogJ3JldGFnZXRpbmc6Y29va2llJywgJ2RpZCc6ICd5YWhvby5jb20nLCAndmFsaWRhdGlvbic6IFRydWV9 (127.0.0.1) 6.69ms

Upvotes: 1

Views: 1515

Answers (2)

leporo
leporo

Reputation: 594

Mybe it's a bit too late, but someone still may stumble upon this question.

When you use an yield statement with tornado.gen.engine and tornado.gen.Task wrappers you delay the execution of a code below the yielding point until the operation (that you've started in the yield gen.Task(..) statement) completes. In most cases you'll use yield statement to wait for a data trasfer completion.

The main point of using pipelines is to pack multiple redis commands to a single request to reduce a number of requests to the Redis server. So when you add a command to the pipeline nothing is sent to the server. There is no need to use yield statements at this point.

When the execute method is called, all commands, previously packed to a pipeline, are sent to the Redis server. It's better to wrap the execute method call with the yield gen.Task() even if you don't want to wait for the request completion to check the results.

A code using the tornado-redis.Client.pipeline method may look like this:

client = Client()
pipe = client.pipeline()
pipe.hset('foo', 'bar', 1)
pipe.expire('foo', 60)

re1, res2 = yield gen.Task(pipe.execute)

or you may use the 'with' statement:

client = Client()
with client.pipeline() as pipe:
    pipe.hset('foo', 'bar', 1)
    pipe.expire('foo', 60)

    res1, res2 = yield gen.Task(pipe.execute)

The pipeline feature uses the Redis transaction functions. You may read about them in this article: http://redis.io/topics/transactions

Upvotes: 3

Nikolay Fominyh
Nikolay Fominyh

Reputation: 9246

In your case it looks, like pipeline function can not be used with gen.engine.

Have you tried to use t = c.pipeline() instead of t = yield tornado.gen.Task(c.pipeline)?

Upvotes: 1

Related Questions