Twisted/Python - Call Method Inside a Protocol From Another Thread

Question

sorry if I got the title wrong, I'm new to Twisted and couldn't really describe my problem that well.

So the problem is, I have an IRC bot based off ircLogBot.py(http://twistedmatrix.com/documents/current/words/examples/ircLogBot.py) which relays messages between IRC and a MySQL database through a PHP page.

It has to load a PHP page every 1 second, parse the content(JSON), loop through it, then post every item to IRC. I have all of that sorted except posting it to IRC.

The reason it's hard is because the loop runs inside another thread (it has to to work) and I don't know how to call msg() from that thread.

This description has probably been really confusing, so take a look at my code. I commented the bit where I want to send my message:

from twisted.words.protocols import irc
from twisted.internet import reactor, protocol, threads
from twisted.python import log
# system imports
import time, sys, json, urllib, urllib2, threading
url = 'http://86.14.76.169/root/scripts/ircbot.php'
urltwo = 'http://86.14.76.169/root/scripts/returnchat.php'
class LogBot(irc.IRCClient):
    try:
        """A logging IRC bot."""
        nickname = "WorldConflictBot"
        def connectionMade(self):
            irc.IRCClient.connectionMade(self)


        def connectionLost(self, reason):
            irc.IRCClient.connectionLost(self, reason)


        # callbacks for events

        def signedOn(self):
            """Called when bot has succesfully signed on to server."""
            self.join(self.factory.channel)

        def joined(self, channel):
            """This will get called when the bot joins the channel."""
            self.action('JackBot', self.factory.channel, 'Joined')
        def privmsg(self, user, channel, msg):
            """This will get called when the bot receives a message."""
            user = user.split('!', 1)[0]
            values = {}
            values = {'type' : 'message',
                      'message' : msg,
                      'username' : user,
                     }
            data = urllib.urlencode(values)
            req = urllib2.Request(url, data)
            response = urllib2.urlopen(req)
            the_page = response.read()
            # Check to see if they're sending me a private message
            if channel == self.nickname:
                msg = "It isn't nice to whisper!  Play nice with the group."
                self.msg(user, msg)
                return

            # Otherwise check to see if it is a message directed at me
            if msg.startswith(self.nickname + ":"):
                msg = "%s: Hey :)" % user
                self.msg(channel, msg)


        def action(self, user, channel, msg):
            """This will get called when the bot sees someone do an action."""
            user = user.split('!', 1)[0]


        # irc callbacks

        def irc_NICK(self, prefix, params):
            """Called when an IRC user changes their nickname."""
            old_nick = prefix.split('!')[0]
            new_nick = params[0]
            values = {}
            values = {'type' : 'nick',
                      'from' : old_nick,
                      'to' : new_nick,
                     }
            data = urllib.urlencode(values)
            req = urllib2.Request(url, data)
            response = urllib2.urlopen(req)
            the_page = response.read() 


        # For fun, override the method that determines how a nickname is changed on
        # collisions. The default method appends an underscore.
        def alterCollidedNick(self, nickname):
            """
            Generate an altered version of a nickname that caused a collision in an
            effort to create an unused related name for subsequent registration.
            """
            return nickname + '^'

    except KeyboardInterrupt:
        LogBotLooper.exit()
        sys.exit()

class LogBotFactory(protocol.ClientFactory):
    """A factory for LogBots.

    A new protocol instance will be created each time we connect to the server.
    """

    def __init__(self):
        self.channel = 'worldconflict'

    def buildProtocol(self, addr):
        p = LogBot()
        p.factory = self
        return p
        l = LogBotLooper()
        l.factory = self
        return l
    def clientConnectionLost(self, connector, reason):
        """If we get disconnected, reconnect to server."""
        connector.connect()

    def clientConnectionFailed(self, connector, reason):
        print "connection failed:", reason
        reactor.stop()

class LogBotLooper(irc.IRCClient):
    def __init__(self):
        i = 0
        lastid = 0
        while 1:       
            time.sleep(1)
            if(i == 0):
                values = {'justlastid': 'true'}
            else:
                values = {'lastid' : lastid}
            data = urllib.urlencode(values)
            req = urllib2.Request(urltwo, data)
            response = urllib2.urlopen(req)
            the_page = response.read()
            if(i == 0):
                lastid = the_page
                i += 1
            else:
                if(the_page != 'error'):
                    jsonpage = json.loads(the_page)
                    for message in jsonpage['messages']:
                        #Need to send the variable `message` to IRC.
                    lastid = jsonpage['highestid']

    def exit(self):
        sys.exit()

if __name__ == '__main__':
    try:
        # initialize logging
        log.startLogging(sys.stdout)

        # create factory protocol and application
        f = LogBotFactory()
        # connect factory to this host and port
        reactor.connectTCP("irc.skyirc.net", 6667, f)
        reactor.callInThread(LogBotLooper)
        # run bot
        reactor.run()
    except KeyboardInterrupt:
        LogBotLooper.exit()
        sys.exit()

Answer 1

If I didn't get wrong message from your post, I have same problem like yours.

http://twistedmatrix.com/documents/10.1.0/core/howto/threading.html

threads.blockingCallFromThread is another answer for this question.

Just replace

#Need to send the variable `message` to IRC.

with

threads.blockingCallFromThread(reactor, irc.send_message, message) 
#I assume you call irc.send_message here

Answer 2

You probably knew this already, but your protocol class should really just stick to event handling and other abstractions over the transport itself. That way you preserve separation of concerns and you have a maintainable framework. In the MVC paradigm, your protocol class is a controller, or maybe even a view, but definitely not a model. Making PHP web service calls probably belongs in the model.

As far as transferring work to other threads (which you will definitely need to do for any blocking I/O such as web service calls), you need:

from twisted.internet import threads, reactor

From the main reactor thread, call threads.deferToThread(mycallable, *args, **kwargs) to have mycallable called from the next available worker thread.

From any worker thread, call reactor.callFromThread(mycallable, *args, **kwargs) to have mycallable called from the main reactor thread.

To transfer work from one worker thread to another, combine the two techniques: reactor.callFromThread(threads.deferToThread, mycallable, *args, **kwargs).

I believe both of these calls return a Deferred object (I know deferToThread does). If you add callbacks to the deferred, those callbacks will execute in the same thread as the original callable. To delegate callback execution to worker threads, use the above techniques within the callbacks. (They don't call it "Twisted" for nothing.)