CODEWITHSUNDEEP
c#htmlwebclient
DealerJoe
DealerJoe

Reputation: 95

C# Fetching the HTML Code of a website does not work

I've been trying to download the html source code of a certain website, to be more precisley from this URL:

http://steamcommunity.com/market/search?q=&category_730_ItemSet[]=any&category_730_TournamentTeam[]=any&category_730_Weapon[]=any&category_730_Quality[]=tag_normal&appid=730#p10_price_desc

When I put this URL into my Webbrowser (Firefox), it takes me to page 10 of the Steam-Market, because at the end of the URL, it says 'p10'. When i replace the 10 with a 12, it takes me to page 12. However, when I'm trying to download the 10th page using this code in C#, 

WebClient webClient = new WebClient();
string pageURL = downloadURL + getURLEnding(currentPage);

string html = webClient.DownloadString(pageURL);
var doc = new HtmlDocument();
doc.LoadHtml(html);

it seems to ignore that bit of the url. No matter what number I replace the 10 with, I'm always getting the same result. Why ?

Upvotes: 0

Views: 102

Answers (1)

Benoit Esnard
Benoit Esnard

Reputation: 2075

You get the same result because Steam's page actually fetches results with AJAX, which means that the content you want isn't in the page you get.

Here is the URL fetched by your browser when you are on the 10th page (the result is in JSON format):

http://steamcommunity.com/market/search/render/?query=&start=90&count=10&search_descriptions=0&sort_column=price&sort_dir=desc&appid=730&category_730_ItemSet%5B%5D=any&category_730_TournamentTeam%5B%5D=any&category_730_Weapon%5B%5D=any&category_730_Quality%5B%5D=tag_normal

Note the start parameter in this URL, which is calculated from the page number.

Upvotes: 4

Related Questions