CODEWITHSUNDEEP
pythontwisted
Anil J
Anil J

Reputation: 53

twisted web get host:port tuple in async response

I am able to make async requests using examples at: http://twistedmatrix.com/documents/current/web/howto/client.html

I can get the async responses inside Protocol.dataReceived() but how do I find out which response is mapped to which request? I would like to map the response I get back to the original host:port/request.

Also, is it safe to use global like I am doing here?

class ClientResponseProtocol(Protocol):

  def __init__(self, whenFinished):
      self.whenFinished = whenFinished

  def dataReceived(self, bytes):
    global BUILD_REVISION

    d = json.loads(bytes)
    bv = d.get("build_revision")

    if bv in BUILD_REVISION:
      BUILD_REVISION[bv] += 1
    else:
      BUILD_REVISION[bv] = 1

  def makeConnection(self, transport):
    pass

  def connectionLost(self, reason):
    self.whenFinished.callback(None)

def handleResponse(r):
  #print("version=%s\ncode=%s\nphrase='%s'" % (r.version, r.code, r.phrase))
  #for k,v in r.headers.getAllRawHeaders():
  #  print(k, v)
  finished = twisted.internet.defer.Deferred()
  r.deliverBody(ClientResponseProtocol(finished))
  return finished

def handleError(reason):
  reason.printTraceback()
  reactor.stop()

def getPage(url):
  d = Agent(reactor).request('GET', url, Headers({'User-Agent': ['user']}), None)
  d.addCallbacks(handleResponse, handleError)
  return d


semaphore = twisted.internet.defer.DeferredSemaphore(BATCH_SIZE)
dl = list()

for server in servers:
  dl.append(semaphore.run(getPage, 'http://%s/server_info.json' % server))

dl = twisted.internet.defer.DeferredList(dl)
dl.addCallbacks(lambda x: reactor.stop(), handleError)

reactor.run()

Upvotes: 1

Views: 343

Answers (2)

Roman Mogylatov
Roman Mogylatov

Reputation: 566

  • First of all, don't use globals. It is evil.
  • Second, Twisted creates Protocol instance for every connection. So all the attributes that you will define in Protocol.__init__() will be "connection scoped".
  • Third, every Protocol instance should be connected with transport, that describes connection (tcp, for example). Transport object contains data that you need, it could be retrieved like this:
def makeConnection(self, transport):
    # for a TransportProxyProducer
    host = transport._producer.getPeer().host
    port = transport._producer.getPeer().port

Here is some links for api docs that may help:

Upvotes: 0

Jean-Paul Calderone
Jean-Paul Calderone

Reputation: 48335

You can pass extra arguments to a Deferred callback:

d.addCallback(f, extra, positional, args, keyword=args)

So, for example:

def report_result(result, request_url):
    ...

url = 'http://%s/server_info.json' % server
d = getPage(url)
d.addCallback(report_result, request_url=url)

Or in a loop:

list_of_results = []
for server in list_of_servers:
    url = 'http://%s/server_info.json' % server
    d = getPage(url)
    d.addCallback(report_result, request_url=url)
    list_of_results.append(d)

all_requests = DeferredList(list_of_results)
...

Separately, you may also want to look at twisted.web.client.Agent to replace your use of getPage.

Upvotes: 1

Related Questions