CODEWITHSUNDEEP
pythonpython-requestssdngrequests
Sindhujit
Sindhujit

Reputation: 51

Python grequests issue

I am trying to make calls to several URLs at the same time using the grequests library for Python. The problem is that I do not quite understand the logic of grequests. Below is a sample (edited version) of my code:-

respArray = []
response = []
sessionvar = requests.Session()
sessionvar.trust_env = False
for each in range(0,len(urls)):
        response.append(grequests.get(urls[each],session=sessionvar,cookies=cookiesArray[each]))
        eachresp = grequests.map(response)
 for r in eachresp:
    respArray.append(r.json())

return respArray

My respArray returns each individual array which was returned from the URLs. When I run that array, it's as if each one is running on its own loop, and not concurrently. I am not getting how can I get it to run concurrently so that I get faster results.

Upvotes: 1

Views: 1389

Answers (1)

enderland
enderland

Reputation: 14185

This code:

for each in range(0,len(urls)):
        response.append(grequests.get(urls[each],session=sessionvar,cookies=cookiesArray[each]))
        eachresp = grequests.map(response)
 for r in eachresp:
    respArray.append(r.json())

Is effectively sending each request sequentially. You send the url and wait for grequests to send it in each loop iteration.

Basically it looks like:

  • Loop iteration, wait for request
  • Loop iteration, wait for request

etc.

You need to follow how their documentation suggests:

# Build a list of unsent requests
requests = (grequests.get(url) for url in urls)
# Send them all at once
results = grequests.map(requests)
return [r.json() for r in results]

This will build your list of unsent requests and then send them all together. In other words:

  • Loop iteration, create unsent request
  • Loop iteration, create unsent request
  • ...
  • Send all requests simultaneously

Upvotes: 1

Related Questions