CODEWITHSUNDEEP
python-3.xsocketszeromqdistributed-systempyzmq
Drak
Drak

Reputation: 86

Can .select() on both traditional and ZeroMQ sockets?

I have a thread that waits for signaling input via ZMQ and Network IO via TCP ( and another doing the same with UDP ).

socket_tcp = socket.socket( socket.AF_INET, socket.SOCK_STREAM )
socket_tcp.connect( ( self.config.tcp_ip, self.config.tcp_port ) )

while True:
    s_ready = zmq.select( [socket_zmq, socket_tcp], [], [] )[0]
        for sock in s_ready:
            # do stuff

However, the TCP socket is never returned as writable ( I made sure it actually gets data using Wireshark ).

The Documentation says I could pass

A zmq.Socket or any Python object having a fileno()
method that returns a valid file descriptor.

and I guess the later is true for socket.socket.

Its the same with UDP.

Am I missing something?

Do I need to handle the ZMQ socket in a separate thread and pass its messages over using ctrl_rcv, ctrl_snd = multiprocessing.Pipe()?

Or can I use a select over both worlds, as I would expect?

Upvotes: 2

Views: 1892

Answers (2)

user3666197
user3666197

Reputation: 1

Do not panic ( yet )

Well, while the recent ZeroMQ API does offer an ZMQ_FD-file-descriptor proxy for touching the ZeroMQ socket instances, these instances are just a tip of a "hidden"-iceberg and behave a way different from any other "common"-socket abstraction, as published here, so one ought better straight forget ZeroMQ to be "something-like-socket"

Best
be warned about at least
the main conceptual differences in the [ ZeroMQ hierarchy in less than a five seconds ] Section.

The powers of the ZeroMQ framework tools are in their concise design, so it is almost sure, it will lose all these unique advantages, if used just "side-by-side" ( in whatever geeky syntax-constructor trick ) with any principally way-simpler ( primitive, if one wishes so ) signalling / messaging device.

If one took all efforts so far to implement a ZeroMQ signalling infrastructure, it would be a nonsense not to use it.

So rather deliver inter-thread messaging again better over ZeroMQ, best using zero-copy smart inproc:// transport-class and avoid any platform-specific overheads a multiprocessing.Pipe() will additionally create just for moving ( ... and replicating ) the already known pieces of data ( ZeroMQ may enjoy an ultra-low latency right due to zero-copy, where applicable ).

Upvotes: -1

Drak
Drak

Reputation: 86

After playing around with asyncio I discovered that zmq.asyncio works perfectly fine with my use case. It makes use of zmq.Poller() (as does zmq.select). But for some reason it just works...

So I ended up using ZMQSelector() with a few additional lines of code:

import selectors
from zmq.asyncio import ZMQSelector

ctx = zmq.Context()
mixed_selector = ZMQSelector()
mixed_selector.register(sock_tcp, selectors.EVENT_READ)
mixed_selector.register(sock_zmq, selectors.EVENT_READ)

while True:
    fd_event_list = mixed_selector.select()
    for key, event in fd_event_list:
        sock = key.fileobj
        # do stuff

Upvotes: 2

Related Questions