CODEWITHSUNDEEP
c#htmlxpathweb-scrapinghtml-agility-pack
Japo Japic
Japo Japic

Reputation: 77

HtmlAgilityPack scraping - extracting specific nodes from html document

Apologies in advance if this has already been answered(if so please point me to right location), I searched here, web, youtube and so on for two days and still haven't founnd an answer.

I would like to extract some data from following url: https://betcity.ru/en/results/sp_fl=a:46;

I am trying to get all event names for the day(1st one is: Ho Kwan Kit/Wong Chun Ting — Fan Zhendong/Xu Xin and all others after it). When I inspect that element I can see this part of html:

<div class="content-results-data__event"><span>Ho Kwan Kit/Wong Chun Ting — Fan Zhendong/Xu Xin</span></div>

I was thinking of getting all div's with class="content-results-data__event" and than get inner text from those div's. Every time I run my code I get zero results. Why am I not getting any nodes when I can see that div's with such class exist and how can I get all events (if I learn how to do that I could get other info which I need from this site). Here is my code (have to say I am fairly new to this). 

public partial class Scrapper : System.Web.UI.Page
{
    protected void Page_Load(object sender, EventArgs e)
    {
        List<string> Events = new List<string>();
        HtmlWeb web = new HtmlWeb();
        HtmlDocument doc = NewMethod(web);
        var Nodes = doc.DocumentNode.SelectNodes(xpath: "//div[@class='content - results - data__event'']").ToList();

        foreach (var item in Nodes)
        {
            Events.Add(item.InnerText);
        }

        GridView1.DataSource = Events;
        GridView1.DataBind();


    }

    private static HtmlDocument NewMethod(HtmlAgilityPack.HtmlWeb web)
    {
        return web.Load("https://betcity.ru/en/results/sp_fl=a:46;");
    }
}

}

Upvotes: 0

Views: 642

Answers (1)

Kent Kostelac
Kent Kostelac

Reputation: 2466

Here is how to get the HTML for one day of matches using Selenium. Rest is HtmlAgilityPack. The site uses self signed certificates so I had to configure the driver to accept self signed certificates. Have fun.

        var ffOptions = new FirefoxOptions();
        ffOptions.BrowserExecutableLocation = @"C:\Program Files (x86)\Mozilla Firefox\firefox.exe";
        ffOptions.LogLevel = FirefoxDriverLogLevel.Default;
        ffOptions.Profile = new FirefoxProfile { AcceptUntrustedCertificates = true };
        var service = FirefoxDriverService.CreateDefaultService();
       var driver = new FirefoxDriver(service, ffOptions, TimeSpan.FromSeconds(120));

        string url = "https://betcity.ru/en/results/date=2017-11-19;"; //remember to update the date accordingly.

        driver.Navigate().GoToUrl(url);
        Thread.Sleep(2000);
        Console.Write(driver.PageSource);

Upvotes: 0

Related Questions