CODEWITHSUNDEEP
pythonmultithreadingmultiprocessingtwistedzeromq
Ground Hog
Ground Hog

Reputation: 303

Python Twisted TCP application - How to prevent incoming message loss by blocking process

I have 10 messages/second(total activity) coming in on TCP from 40 clients. I need to take each message and do a 5 second process (look up a webservice, do some DB queries and finally write the results to the DB).

How do I separate messages coming in from the slow 5 second process? Also I might receive another message from a client while already processing a message for that client. I NEVER want to lose a message.

Upvotes: 0

Views: 506

Answers (2)

Jean-Paul Calderone
Jean-Paul Calderone

Reputation: 48335

With Twisted, the answer is to simply do what you want to do:

from twisted.python.log import err
from twisted.internet.protocol import Protocol

class YourProtocol(Protocol):
    ...
    def messageReceived(self, message):
        d = lookupWebService(message)
        d.addCallback(queryDatabase)
        d.addCallback(saveResults)
        d.addErrback(err, "Servicing %r failed" % (message,))

You can find APIs for interacting with web services in twisted.web.client (presuming "web services" are things you talk to using an HTTP client). You can find APIs for interacting with some SQL database servers in twisted.enterprise.adbapi. You can find APIs for interacting with other kinds of databases with a little googling.

Upvotes: 1

raffian
raffian

Reputation: 32076

Distribute tasks in parallel using Divide and Conqueur.

Lots of python examples illustrating this approach, read up here:

enter image description here

You can also distribute tasks using a ROUTER/DEALER proxy. Messages arriving at the proxy are fair-queued and distributed amongst downstream workers, no back chatter; this approach may better fit your needs.

enter image description here

Upvotes: 0

Related Questions