CODEWITHSUNDEEP
c#web-scrapingdotnet-httpclient
Twenty
Twenty

Reputation: 5871

Get HTML Code from a website after it completed loading

I am trying to get the HTML Code from a specific website async with the following code:

var response = await httpClient.GetStringAsync("url");

But the problem is that the website usually takes another second to load the other parts of it. Which I need, so the question is if I can load the site first and read the content after a certain amount of time.

Sorry if this question already got answered, but I didn't really know what to search for.

Thanks, Twenty

Edit #1

If you want to try it yourself the URL is http://iloveradio.de/iloveradio/, I need the Title and the Artist which do not immediately load.

Upvotes: 18

Views: 9465

Answers (5)

Alexander O'Mara
Alexander O'Mara

Reputation: 60507

The thing to understand here is that when you read the response from the URL, all you will ever get is the raw response, in this case the HTML source code the server replied with.

Unlike what you might see in your browser's DOM Inspector developer tools, you will only get the original HTML source from the page (what you might see in the "Page Source" developer tool) which will not include any dynamically created content (JavaScript) or loaded content (like iframes).

So you aren't getting what you see here in the DOM Inspector:

enter image description here

You are getting what you see here in the Page Source (View > Developer > View Source in Chrome):

enter image description here

You can't wait for that other content to load because it will never load since that HTML content isn't being parsed or rendered like a browser would.

You have several options available though:

  • See if the website has an API you can use
  • Determine where that content you want is actually loaded from, and make another/different HTTP request to that content (the Network Panel is helpful here)
  • Use a headless browser to programmatically load the page and dynamically read the page contents (this will add a lot of overhead, and should probably be avoided if possible)

Upvotes: 3

Derviş Kayımbaşıoğlu
Derviş Kayımbaşıoğlu

Reputation: 30545

You are on the wrong direction. The referenced site has playlist api which returns json. you can get information from :

http://iloveradio.de/typo3conf/ext/ep_channel/Scripts/playlist.php

Edit: Chome Inspector is used to find out Playlist link

enter image description here

Upvotes: 12

Ashkan Mobayen Khiabani
Ashkan Mobayen Khiabani

Reputation: 34152

If there are things that load after, it means that they are generated by javascript code after page load (an ajax request for example), so no matter how long you wait, it won't have the content you want (because they are not in the source code when it loads).

Easy way to do it:

Use a WebBrowser and when DocumentCompleated event triggers wait till the element you want appears.

The Right Way:

find the javascript yourself and trigger it yourself (easy to say, hard to do).

Upvotes: 4

Fagun
Fagun

Reputation: 79

I have checked out the website, data is loaded by javascript. You only can get the html using httpClient.GetStringAsync("url");. As far as I know, there is no luck to get the elements what is manipulate by browser.

Upvotes: 0

hardkoded
hardkoded

Reputation: 21597

You could use Puppeteer-Sharp:

await new BrowserFetcher().DownloadAsync(BrowserFetcher.DefaultRevision);
using (var browser = await Puppeteer.LaunchAsync(new LaunchOptions { Headless = false }))
using (var page = await browser.NewPageAsync())
{
    await page.SetViewportAsync(new ViewPortOptions() { Width = 1280, Height = 600 });
    await page.GoToAsync("http://iloveradio.de/iloveradio/");
    await page.WaitForSelectorAsync("#artisttitle DIV");
    var artist = await page.EvaluateExpressionAsync<string>("$('#artisttitle DIV')[0].innerText");
    Console.WriteLine(artist);
    Console.ReadLine();
}

Upvotes: 4

Related Questions