CODEWITHSUNDEEP
pythonmultithreadingasynchronoustwisted
DenverCoder9
DenverCoder9

Reputation: 151

Twisted callRemote

I have to make remote calls that can take quite a long time (over 60 seconds). Our entire code relies on processing the return value from the callRemote, so that's pretty bad since we're blocking on IO the whole time despite using twqisted + 50 worker threads running.

We currently use something like

result = threads.blockingCallFromThread(reactor, callRemote, "method", args)

and get the result/go on, but as its name says it's blocking the event loop so we cannot wait for several results at the same time.

THere's no way I can refactor the whole code to make it asynchronous so I think the only way is to defer the long IO tasks to threads.

I'm trying to make the remote calls in threads, but I can't find a way to get the result from the blocking calls back. The remoteCalls are made, the result is somewhere but I just can't get a hook on it.

What I'm trying to do currently looks like 

reactor.callInThread(callRemote, name, *args, **kw)

which returns a empty Deferred (why ?).

I'm trying to put the result in some sort of queue but it just won't work. How do I do that ?

Upvotes: 2

Views: 463

Answers (1)

Viach Kakovskyi
Viach Kakovskyi

Reputation: 1517

AFAIK, blockingCallFromThread executes code in reactor's thread. That's why it doesn't work as you need.

If I understand you properly, you need to move some operation out off reactors thread and get the result into reactors thread.

I use approach with deferToThread for the same case. Example with deferreds:

import time
from twisted.internet import reactor, threads

def doLongCalculation():
    time.sleep(1)
    return 3

def printResult(x):
    print x

# run method in thread and get result as defer.Deferred
d = threads.deferToThread(doLongCalculation)
d.addCallback(printResult)
reactor.run()

Also, you might be interested in threads.deferToThreadPool.

Documentation about threading in Twisted.

Upvotes: 2

Related Questions