CODEWITHSUNDEEP
pythonpython-requestsgrequests
ZhaiShang
ZhaiShang

Reputation: 133

part of the requests delay the whole process in grequests

I have 600 urls to request, when I use grequests, I found that sometimes it finishes so fast within 10 secs, but sometimes just stuck there(can't reach the statement of printing 'done').

Here is my code:

l=[]
urls=[...]
reqs=[grequests.get(i) for i in urls]

rs=grequests.map(reqs)
print('done')

Is it because majority of the requests are already finished within 10 secs(with status 200), just a few requests are still waiting for response? I am using Pycharm, in the variable monitor window, I can indeed see that lots of requests already get 200 status. How can I fix this problem? Is there any params that I can set to limit the maximum request time for each request, therefore if some requests exceed a certain time, it will return anyway.

Upvotes: 0

Views: 441

Answers (1)

Alexey
Alexey

Reputation: 54

Just set the timeout:

import grequests

l=[]
urls = [ ]
reqs=[grequests.get(i, timeout=1) for i in urls] # timeout in sec

rs=grequests.map(reqs)
print(rs)

Upvotes: 0

Related Questions