CODEWITHSUNDEEP
ramazon-web-serviceszeromqaws-batch
landau
landau

Reputation: 5851

Interprocess communication on AWS Batch

Is it possible to use ZeroMQ for interprocess communication between a job on AWS Batch and a local parent process? If so, is there a simple example I can refer to? I am having trouble finding documentation on this.

I am asking with hopes that the clustermq R package will someday support an AWS Batch backend: https://github.com/mschubert/clustermq/issues/208. clustermq can already send R function calls to workers on traditional job schedulers like SLURM and communicate with the workers using ZeroMQ.

Upvotes: 1

Views: 220

Answers (1)

user3666197
user3666197

Reputation: 1

Q : "Is it possible to use ZeroMQ for interprocess communication between a job on AWS Batch and a local parent process?"

ZeroMQ is a can-do toolbox, so the only problem will be with the Cloud-owner dictated policies (plus the grooming-factor bottlenecks, as these always has been the new kind of singularity-related issues, a SPoF if someone wants...)

Distributed-computing per-se has no other problems with ZeroMQ.

A PoC Example

We may instantiate this code on all remote-nodes and let them run & communicate with your localhost, sure if not blocked by Cloud-owner's set of policies and if you can let the receiving side survive the Cloud-initiated tsunami ;o)

Connectivity was set so that remote-senders code "knows" the localhost's either public IP address, or similarly motivated SSH-tunneling meta-plane, that mediates interconnections through the SSH-port-forwarding-tunnels, thus allowing all the Cloud remote-senders (if not blocked by the Cloud-owner...) actually reach to localhost private IP address, as if they were in the same private LAN-segment.

( a Python code provided for the ease of cross-porting )

A localhost-receiver mock-up example :

import zmq;         nIOthreads = 4 ### traffic dependent, given an expected ingress Tsunami
aCtx = zmq.Context( nIOthreads )
aSub = aCtx.socket( zmq.XSUB )
aSub.bind( "tcp://<_aPublicIpADDRESS_or_SSL_tunnelled_privateIpADDRESS_>:<_port#_>" )
aSub.setsockopt( zmq.LINGER, 0 )
aSub.setsockopt( zmq.CONFLATE, 0 )
aSub.setsockopt( zmq.SUBSCRIBE, "" )
...
while True:
      try:
           ret = aSub.recv( zmq.NOBLOCK )
           if ret:
                 print( "GOT: {0:}".format( repr( ret ) ) )

      except:
           print( "EXC: will terminate" )
           break

      finally:
           aSub.close()
           aCtx.term()

print( "FIN:  did terminate and process will sysexit." )

A Cloud-remote sender mock-up example :

import socket
import zmq;         nIOthreads = 1 ### beware of the grooming-factor effects on traffic
aCtx = zmq.Context( nIOthreads )
aPub = aCtx.socket( zmq.XPUB )
aPub.connect( "tcp://<_aPublicIpADDRESS_or_SSL_tunnel_to_privateIpADDRESS_>:<_port#_>" )
aPub.setsockopt( zmq.LINGER, 0 )
aPub.setsockopt( zmq.CONFLATE, 0 )
...
aCloudHostMsgN = 1
aCloudHostNAME = socket.gethostname()
aCloudHostIP   = socket.gethostbyname( aCloudHostNAME )
aMsgMASK       = ( "msg[{0:_>9d}]"
                 + "_from_{0:_>20s}_ip_{1:_>20s}".format( aCloudHotsNAME,
                                                          aCloudHostIP
                                                          )
                   )
while True:
      try:
           aPub.send( aMsgMASK.format( aCloudHostMsgN ) )
           pass;                       aCloudHostMsgN++

      except:
           print( "EXC: will terminate" )
           break

      finally:
           aPub.close()
           aCtx.term()

print( "FIN:  did terminate and process will sysexit." )

Upvotes: 2

Related Questions