CODEWITHSUNDEEP
google-app-engineasynchronous
Marco Yammine
Marco Yammine

Reputation: 323

GAE Standard Async Fetch not working

I'm following the docs and yet it appears the requests are still being made synchronously.

https://cloud.google.com/appengine/docs/standard/python/issue-requests

Here is my code:

rpcs = []
for url in urls:
    rpc = urlfetch.create_rpc()
    urlfetch.make_fetch_call(rpc, url)
    rpcs.append(rpc)
result = []
for rpc in rpcs:
    result.append(rpc.get_result().content)
return result

I did some profiling and compared using requests.get and they both take exactly the same amount of time.

The urls i'm fetching are from different sites so I'm sure that I don't have concurrent limitations on the server side.

Running on GAE Standard, Python 2.7

Upvotes: 0

Views: 113

Answers (1)

Marco Yammine
Marco Yammine

Reputation: 323

I got it working but for some reason only with callbacks. Also It only works on production and not on local env. :D. Here is the working code:

from google.appengine.api import urlfetch
import functools


class ClassName(object):

    responses = []

    def fetch_concurrent_callback(self, rpc):
        response = rpc.get_result()
        json_response = json.loads(response.content)
        self.responses.append(json_response)

    def fetch_concurrent(self, urls):
        rpcs = []
        for url in urls:
            rpc = urlfetch.create_rpc()
            rpc.callback = functools.partial(self.fetch_concurrent_callback, rpc)
            urlfetch.make_fetch_call(rpc, url)
            rpcs.append(rpc)
        for rpc in rpcs:
            rpc.wait()
        return self.responses

Upvotes: 1

Related Questions