CODEWITHSUNDEEP
ajaxseleniumweb-scraping
Abdul Ali
Abdul Ali

Reputation: 1937

Browserless web scraping of ajax page

Have tried using Selenium after reading some tutorial for web scraping ..

The aim is to web/screen scrape a page that loads the required data after an Ajax call when (this ajax call made after Initial page load)..

The second aim is to run Selenium code in the background (not opening any browser) to allow loading the page (including the Ajax call) , retrieve the Final HTML and perform required processing locally ..

the code till now is as follows (code tutorial from http://www.geekonweb.com/net/web-page-scraping-using-selenium-and-net/)

public ActionResult Index()
    {
        //--
        //Below path should contain IEDriverServer.exe
        var chrome = new ChromeDriver(@"file path");
        chrome.Url = @"<url>";

        chrome.Navigate();

        //extract the html
        //var retval = ie.ExecuteScript("return document.body.outerHTML");

        string result = chrome.PageSource;


        return View();
    }

currently have not been able to find a way to run Selenium Silently (without GUI). kindly assist if that can be done.

Secondly kindly tell that how can Selenium be told to wait for the Ajax call to finish and then retrieve the data.

regards,

Upvotes: 1

Views: 2631

Answers (1)

Benjamin Gruenbaum
Benjamin Gruenbaum

Reputation: 276296

Here is a question on how to wait until an element is present. This is done to wait for the AJAX.

Here is a question on weather it's possible to run selenium headless.

Upvotes: 1

Related Questions