CODEWITHSUNDEEP
nginxweb-crawler
brisingr1921
brisingr1921

Reputation: 1

How do I find the number of requests I can make to a website before nginx blocks me

If I am trying to crawl and pull data from a website, I should know the number of requests I can make per hour before I find 403 forbidden error. Is there a way I can find out that rate for a website?

Upvotes: 0

Views: 66

Answers (1)

Rahul Soni
Rahul Soni

Reputation: 4968

No, every server has his own limits. There can be servers that wouldn't mind you being around and giving them as many hits as you like.

http://nginx.org/en/docs/http/ngx_http_limit_req_module.html

Also, in general it is not a good idea to do what you want to do.

As a side note, there are websites like Twitter, etc. that allows you to make n number of requests per day, per hour, etc. If such is the case, it is typically documented in the website's developer documents.

Upvotes: 0

Related Questions