CODEWITHSUNDEEP
python-2.7zeromqpyzmq
Joel
Joel

Reputation: 6107

ZMQ IOLoop instance write/read workflow

I am having a weird system behavior when using PyZMQ's IOLoop instance:

def main():
    context = zmq.Context()
    s = context.socket(zmq.REP)
    s.bind('tcp://*:12345')
    stream = zmqstream.ZMQStream(s)
    stream.on_recv(on_message)

    io_loop = ioloop.IOLoop.instance()
    io_loop.add_handler(some_file.fileno(), on_file_data_ready_read_and_then_write, io_loop.READ)
    io_loop.add_timeout(time.time() + 10, another_handler)
    io_loop.start()

def on_file_data_ready_read_and_then_write(fd, events):
   # Read content of the file and then write back
   some_file.read()
   print "Read content"
   some_file.write("blah")
   print "Wrote content"

def on_message(msg):
   # Do something...
   pass

if __name__=='__main__':
    main()

Basically the event loop listens to zmq port of 12345 for JSON requests, and reads content from a file when available (and when it does, manipulate it and wrties to it back. Basically the file is a special /proc/ kernel module that was built for that).

Everything works well BUT, for some reason when looking at the strace I see the following:

...
1. read(\23424) <--- Content read from file
2. write("read content")
3. write("Wrote content")
4. POLLING
5. write(\324324) # <---- THIS is the content that was sent using some_file.write()
...

So it seems like the write to file was not done in the order of the python script, but the system call of write to that file was done AFTER the polling, even though it should have been done between lines 2 and 3.

Any ideas?

Upvotes: 3

Views: 582

Answers (1)

David
David

Reputation: 18281

Looks like you're running into a caching problem. If some_file is a file like object, you can try explicitly calling .flush() on it, same goes for ZMQ Socket which can hold messages for efficiency reasons as well.

As it stands, the file's contents are being flushed when the some_file reference is garbage collected.

Additional:

use the context manager logic that newer versions of Python provide with open()

with open("my_file") as some_file:
     some_file.write("blah")

As soon as it finishes this context, some_file will automatically be flushed and closed.

Upvotes: 1

Related Questions