CODEWITHSUNDEEP
c#html-agility-pack
dimker
dimker

Reputation: 19

C# HtmlAgilityPack timeout before download page

I want parse site https://russiarunning.com/events?d=run on C# with htmlagilitypack I'm try this make 

string url = "https://russiarunning.com/events?d=run";
var web = new HtmlWeb();
var doc = web.Load(url);

But I got a problem - content on site loading with timeout ~1000ms therefore, when using the web.Load (url) I download the page without content.

How make timeout before download page with htmlagilitypack ?

Upvotes: 0

Views: 1753

Answers (2)

Gauravsa
Gauravsa

Reputation: 6514

You can simply do this:

string url = "https://russiarunning.com/events?d=run";
var web = new HtmlWeb();
web.PreRequest = delegate(HttpWebRequest webReq)
{
 webReq.Timeout = 4000; // number of milliseconds
 return true;
};

var doc = web.Load(url);

More on Timeout property: https://learn.microsoft.com/en-us/dotnet/api/system.net.httpwebrequest.timeout?view=netframework-4.7.2

Upvotes: 2

Akhatarali Ansari
Akhatarali Ansari

Reputation: 70

Try this...

Create one class as below :

public class WebClientHelper : WebClient
{
    protected override WebRequest GetWebRequest(Uri address)
    {
        HttpWebRequest request = base.GetWebRequest(address) as HttpWebRequest;
        request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
        return request;
    }
}

and use as below:

  var data = new Helpers.WebClientHelper().DownloadString(Url);
  var htmlDocument = new HtmlDocument();
  htmlDocument.LoadHtml(data);

Upvotes: 2

Related Questions