Python - share sockets between processes

Question

Is it possible to share socket objects between 2 running processes?

To simplify the issue, assume that we have two files s1.py and s2.py. Both are tcp servers listening on different ports.

s1.py body

from twisted.internet import reactor, protocol

CLIENTS = []

class Echo(protocol.Protocol):
    def connectionMade(self):
        CLIENTS.append(self)

    def dataReceived(self, data):
        self.transport.write(data)

    def connectionLost(self):
        CLIENTS.remove(self)

def main():
    factory = protocol.ServerFactory()
    factory.protocol = Echo
    reactor.listenTCP(8000, factory)
    reactor.run()

main()

and s2.py

from twisted.internet import reactor, protocol

class Echo(protocol.Protocol):

    def dataReceived(self, data):
        for socket in list_of_serialized_redis_CLIENTS_socket_object:
           socket.transport.write(data)

def main():
    factory = protocol.ServerFactory()
    factory.protocol = Echo
    reactor.listenTCP(8001, factory)
    reactor.run()

main()

Is it possible to share CLIENTS from s1.py with s2.py? Maybe there is some way to serialize CLIENTS and store it in redis or so.

Answer 1

Using socket.share() might not be as straight forward as you might think. First off, it's a Windows exclusive feature and Windows is the one platform that gets the least attention by Twisted. Getting access to the low-level socket object in Twisted can be tricky because there are a lot of things happening behind the scenes and might cause strange outcomes if changed. So in other words, it's certainly possible to go this route, but it's not recommended.

If you're open to new solutions, I think an RPC-style server will solve your issue. My thinking is, since the connection objects live in s1, simply create a remote process that will "do stuff" to those connections. Other processes can connect to that server and execute functions on objects local to s1. Twisted's Perspective Broker is a ready-made solution you could leverage.

s1.py

from twisted.internet import reactor, protocol, endpoints
from twisted.spread import pb

class Echo(protocol.Protocol):
    def connectionMade(self):
        self.factory.client_set.add(self)

    def connectionLost(self, reason):
        self.factory.client_set.remove(self)

class Remote(pb.Root):
    def __init__(self, client_set):
        self.client_set = client_set

    def remote_s1_write(self, data):
        msg_tmpl = 'From s1...{0}'
        for client in self.client_set:
            response = msg_tmpl.format(data).encode('utf-8')
            client.transport.write(response)

def main():
    client_set = set()      # container will be shared between the 2 servers

    # setup Echo server
    factory = protocol.ServerFactory()
    factory.protocol = Echo
    factory.client_set = client_set
    echo_server = endpoints.TCP4ServerEndpoint(reactor, interface='localhost', port=8000)
    echo_server.listen(factory)

    # setup PB server
    pb_root = Remote(client_set)
    pb_factory = pb.PBServerFactory(pb_root)
    pb_server = endpoints.TCP4ServerEndpoint(reactor, interface='localhost', port=7999)
    pb_server.listen(pb_factory)

    reactor.run()

main()

s2.py

from twisted.internet import reactor, protocol, endpoints
from twisted.spread import pb

class Echo(protocol.Protocol):
    def dataReceived(self, data):
        remote = self.factory.pb_factory.getRootObject()
        remote.addCallback(lambda obj: obj.callRemote('s1_write', data))
        remote.addCallback(lambda echo: 's1 said: {0}'.format(data))
        remote.addErrback(lambda reason: 'error: {0}'.format(str(reason.value)))
        remote.addCallback(lambda response: print(response))
        self.transport.write(data)

def main():
    # connect to s1's PB server
    pb_factory = pb.PBClientFactory()
    pb_connection = endpoints.TCP4ClientEndpoint(reactor, host='localhost', port=7999)
    pb_connection.connect(pb_factory)

    # setup Echo server
    factory = protocol.ServerFactory()
    factory.protocol = Echo
    factory.pb_factory = pb_factory
    echo_server = endpoints.TCP4ServerEndpoint(reactor, interface='localhost', port=8001)
    echo_server.listen(factory)

    reactor.run()

main()

I've taken the liberty and updated the syntax. Try this code and see if it works and ask if you have any issues.