CODEWITHSUNDEEP
pythonjsonbase64zeromqencode
Aakash Basu
Aakash Basu

Reputation: 1767

How to send a Python dictionary consisting of ndarray and None using ZeroMQ PUB/SUB?

I'm trying to pass a python dictionary of 3 images (stored as ndarray) using ZeroMQ to pass it to another program as a consumer and parse back the data to original form. Followed three ways, but couldn't achieve success in anyone of the ways.

Below is the sample minimal reproduced code:

import pickle
import zmq

# Adding ZMQ Context
def zmq_context():
    # Creating ZMQ context starts
    context = zmq.Context()
    footage_socket = context.socket(zmq.PUB)
    footage_socket.connect('tcp://localhost:5002')
    return footage_socket
    
wagon_dtl, ctr1_dtl, ctr2_dtl = NdArrays of images
socket_ctx = zmq_context()

# Trying two different ways of formatting the image before creating the dict, the below approach works for all three ndarrays

# 1st way
wagon_dtl = image_np_save # image_np_save is the original image

# 2nd way (this I tried because an ndarray isn't JSON serializable)
encoded, buffer = cv2.imencode('.jpg', image_np_save)
wagon_dtl = base64.b64encode(buffer)
            

if cond == "fulfilled":
    full_wgn_dict = {"wagon_dtl": wagon_dtl, "ctr1_dtl": ctr1_dtl, "ctr2_dtl": ctr2_dtl}
    
    # 1st way
    dict_as_b64text = base64.b64encode(full_wgn_dict)
    socket_ctx.send(dict_as_b64text)
    
    # 2nd way
    myjson = json.dumps(full_wgn_dict)
    socket_ctx.send(myjson)
    
    # 3rd way
    dict_as_text = pickle.dumps(full_wgn_dict).encode('base64', 'strict')
    socket_ctx.send(dict_as_text)

How to solve this?

I've followed these Q/As while working on this solution: 1, 2, 3, 4, 5

Upvotes: 0

Views: 670

Answers (1)

user3666197
user3666197

Reputation: 1

Q : "How to send a Python dictionary consisting of ndarray and None using ZeroMQ PUB/SUB?"

Easy, one may best use the ready-made .send_pyobj()-method for doing right this.

The sending side,
the PUB shall be doing a call to the socket.send_pyobj( full_wgn_dict )-method, and that's basically all on this side.

A receiving side,
each of the potential SUB-s shall reuse the .recv_pyobj()-method.

Yet all the SUB-s have to also do one more step, to actively subscribe to receive any message at all.

For details on socket.setsockopt( zmq.SUBSCRIBE, "" ) see the ZeroMQ documented API, or do not hesitate to sip from many examples here.

Some additional tricks (not needed for trivial dict-s) may help with the pickle-phase of the SER/DES stage. Yet these go way beyond of the scope of this Question, and may introduce advantages in controlled environments but problems in open, uncontrolled environments, where you lack zero chances to meet the required prerequisites - in my apps, I prefer to use import dill as pickle for having way higher robustness of the pickle.dumps()-SER/DES processing of objects and many more advances, like storing a full-session snapshot. Credits go to @MikeMcKearns

Feel free to re-read the documentation present for all syntax-related details in the __doc__ strings:

>>> print zmq.Socket.send_pyobj.__doc__
Send a Python object as a message using pickle to serialize.

        Parameters
        ----------
        obj : Python object
            The Python object to send.
        flags : int
            Any valid flags for :func:`Socket.send`.
        protocol : int
            The pickle protocol number to use. The default is pickle.DEFAULT_PROTOCOL
            where defined, and pickle.HIGHEST_PROTOCOL elsewhere.
        
>>> print zmq.Socket.recv_pyobj.__doc__
Receive a Python object as a message using pickle to serialize.

        Parameters
        ----------
        flags : int
            Any valid flags for :func:`Socket.recv`.

        Returns
        -------
        obj : Python object
            The Python object that arrives as a message.

        Raises
        ------
        ZMQError
            for any of the reasons :func:`~Socket.recv` might fail

Upvotes: 1

Related Questions