CODEWITHSUNDEEP
c#web-crawler
actkatiemacias
actkatiemacias

Reputation: 1473

Best way to add a delay in C#

I am currently building a web crawler and want to add a delay on request so I don't hit web servers to hard.

What is the most appropriate way to add this delay?

Also, what is an appropriate amount of delay for a crawler? I was thinking a .5-1 second.

Thanks

Upvotes: 0

Views: 541

Answers (1)

alex
alex

Reputation: 12654

You can use

Thread.Sleep(TimeSpan.FromSeconds(1))

Search engines may use larger delays of 10-15 or even 60 seconds. And polite crawlers respect the "Crawl-delay" that site provided in robots.txt. For details, take a look at http://en.wikipedia.org/wiki/Web_crawler#Politeness_policy

Upvotes: 7

Related Questions