CODEWITHSUNDEEP
pythonhtml-parsinghttp-status-code-429grequests
Mop
Mop

Reputation: 53

How to avoid 429 error when using grequests?

I want to make a parser using "grequests", but as expected, due to many requests, i get an error 429.

import grequests


sites = [f"https://fantlab.ru/work{i}" for i in range(1, 10)]

response = (grequests.get(url, timeout=5)  for url in sites)
resp = grequests.map(response)

print(resp)

[<Response [200]>, <Response [200]>, <Response [200]>, <Response [404]>, <Response [429]>...]

Upvotes: 1

Views: 497

Answers (1)

Jaffa
Jaffa

Reputation: 319

A 429 (Too many requests) status code is returned when you are sending too many requests. You could try showing down the requests (using time.sleep(n) from the time module, or see if the owner of the server can increase your quota.

Upvotes: 1

Related Questions