CODEWITHSUNDEEP
pythonmultithreadingpython-2.7zeromqpyzmq
Jason S
Jason S

Reputation: 189906

running a 2nd zmq.eventloop.ioloop

I want to create a PyZMQ eventloop in a background thread, and have it work correctly with both standalone Python scripts and IPython scripts. (IPython uses PyZMQ eventloops located in the main thread, so this is causing me problems and why I want to start a private ioloop in a background thread.)

I want to run code in Thread A while having the PyZMQ eventloop handle received data from a socket in Thread B. There are times in Thread A where I will need to wait for an event set in Thread B.

How can I get this to work? There seems to be something wrong if I try in IPython:

from zmq.eventloop import ioloop
import threading

class IOBackgroundLoop(object):
    def __init__(self):
        self._loop = None
        self._thread = threading.Thread(target=self.run)        
        self._thread.daemon = True
        self._started = threading.Event()
    @property
    def loop(self):
        return self._loop
    def run(self):
        self._loop = ioloop.IOLoop()
        self._loop.initialize()
        self._loop.make_current()
        self._started.set()
        self._loop.start()
    def start(self):
        self._thread.start()
        self._started.wait()

bkloop = IOBackgroundLoop()
bkloop.start()
for loop in [bkloop.loop, ioloop.IOLoop.instance()]:
    print "%s running: %s" % (loop, loop._running)

This prints out two separate instances of IOLoop, but if I go to use it, it doesn't seem to work. I can't think of a small example program to demonstrate this; I've tried with a timeout function:

import time

def print_timestamp(key):
    print "%s: %s" % (time.time(), key)

for loop in [bkloop.loop, ioloop.IOLoop.instance()]:
    loop.add_timeout(bkloop.loop.time() + 1.0, lambda: print_timestamp("hi from %s" % loop))
    print_timestamp("here")
    time.sleep(2.0)
    print_timestamp("there")

and I get this as a result (no "hi":

1412889057.68: here
1412889059.68: there
1412889059.68: here
1412889061.68: there

Then when I hit another shift+Enter, I get

1412889061.68: hi from <zmq.eventloop.ioloop.ZMQIOLoop object at 0x000000000467E4E0>

which is the IOLoop object from the main thread, but my private instance IOLoop never prints hi.

What could I be doing wrong?

Upvotes: 1

Views: 351

Answers (1)

Jason S
Jason S

Reputation: 189906

Argh, I just noticed this in the tornado docs:

Note that it is not safe to call add_timeout from other threads. Instead, you must use add_callback to transfer control to the IOLoop's thread, and then call add_timeout from there.

It also appears as though the zmq.eventloop.zmqstream needs to be setup in the same thread as the ioloop for it to work properly.

Upvotes: 1

Related Questions