CODEWITHSUNDEEP
socketszeromq
Michael Schubert
Michael Schubert

Reputation: 2796

How does ZeroMQ queue and send queued messages?

I am the author of an R package (clustermq0) that distributes function calls on HPC schedulers using using the ZeroMQ bindings (rzmq). I have used a simple combination of REQ/REP sockets, with the workers requesting first the common data for all tasks (the function to call and constant arguments), and then data for each call they should evaluate from the master. This worked well so far, because running the computations is usually an order of magnitude slower than sending and receiving the data.

One issue however is that the common data can have several hundred MB in size, while the iterated data is generally small. It can thus happen that the master will be busy sending a huge chunk of common data and can not send iterated data at the same time. Because of this, there is a noticeable delay when starting up a distributed computation.

However, this may not be caused by the actual sending but rather preparing the message. The documentation states:

ZeroMQ does not send the message (single or multipart) right away, but at some indeterminate later time.

So I'm wondering:

Note that I'm looking for an answer from the design rationale of ZeroMQ and not a comment that I can benchmark.

Some clarifications below:

0 This is not meant to be implemented in the theoretically most efficient way, but rather using functions that rzmq provides. The goal is to improve upon packages that store everything on NAS and retrieve it from there (which is quite a low bar). This is a side project and I am not a systems engineer (and I'm not proficient in low-level ZeroMQ). I am benchmarking both overhead and real-world (a.k.a. my actual work) examples, but that hasn't made it into the docs yet.

1 Assume the cases (TCP): one REP master and n REQ clients; one ROUTER master and n REQ clients; PUSH/PULL as alternative approach. Is there another way to interface with this apart from using different sockets (probably not from high-level bindings like rzmq but pointing me to relevant low-level documentation would also help; I haven't found this information in the user guide)

2 What I mean by this is that if I connect REQ clients to a ROUTER master, I manage the envelopes myself (and have to send the id and empty frame manually), but this does not change the code that ZeroMQ uses under the hood to send messages. Or does it? Where is this documented? (I couldn't find it in the user guide)

3 A valid answer for this would be that bottlenecks are memory copy for initializing the message in the main thread and then sending the messages to one client after the other in a separate thread, not blocking the main (if that is the case, or whatever actually happens to the message)

Upvotes: 1

Views: 2338

Answers (1)

user3666197
user3666197

Reputation: 1

1 ) showing Zero-code means any answer may be just at very high-level

The trailing note:

Note that I'm looking for an answer from the design rationale of ZeroMQ and not a comment that I can benchmark.

did not help either.

So, let's start point after point:

Is ZeroMQ sending ... one after the other or in parallel?

  • ZeroMQ Context-instance is the master to answer this. It depends on how your code has instantiated the data-pumping engines. With Zero-code posted, no one can tell you either or.

Would this make a difference or is it likely negligible?

  • Be sure it makes a difference, a big one.

Is there a way to influence this?

  • Yes, there are several ways to influence this. Depends on your code. Depends on your advertised HPC/Cluster Project End-to-End architecture. As far as my experience goes, there is no universal one-size-fits-all or any cheap ( or free ) magic-wand. Best to use for your Project a pool of indepth knowledge about real-time system scheduling ( and benchmark, benchmark, benchmark -- if you want to withhold the Git-posted promise of a superior performance this package ought both achieve in tests and sustain to exhibit in real-world deployments ).

switching from REP to ROUTER here will not change anything.

Is this correct?

  • No one serious will ever tell you without posting your architecture, implementation rationale and the code itself. Is 42 correct or not? Who knows?!? ( Sure, except the mice and, maybe, the Marvin. ( All relevant facts and details to be found in the Hitchhiker's Guide -- the idea was borrowed from there ) )

Is the main delay likely caused by what happens before, i.e. copying big chunks of memory for creating the message object? ( I already serialize only once )

  • The answer ( even using a probabilistic view ) is 100% hidden in your code. ZeroMQ Context, if configured properly, will not add any remarkable delay just on its own. The process is well documented in ZeroMQ API documentation, so if one tries to marshall a 1kB, 1 MB or "several hundreds MB" BLOB to a .send()-method, one ought know pretty well the reasons for doing this in his/her own way.

In this case I would want to interface with ZeroMQ message objects without copying

  • Well, this is always a preferred way how to dispatch data inside ZeroMQ. Be also warned, that the Zero-copy maxim does not cover the O/S-kernel data-buffers manipulations, so a serious Project plan ought account for realistic operations ( quantum entanglement as a mass-less instant signalling at infinite distance in zero-time or teleportation does not work in our current O/S-kernels, so rather bear in mind the current known silicon and hardware principles )

Upvotes: 2

Related Questions