CODEWITHSUNDEEP
htmlrubyweb-scrapingnokogirimechanize
Matt
Matt

Reputation: 3180

Any way to deal with dynamically loaded content with Mechanize?

I'm trying to get some game listings from Steam, but it hasn't been working. Basically, the page by default loads the first page of "Relevant," the URL of which looks like this:

http://store.steampowered.com/search/?auction=1#sort_by=_ASC&page=1&auction=1

If I change the URL to &page=2, using Inspect Element on the page shows the updated list, however viewing the page source (in both Firefox and Nokogiri) still shows the default content.

No matter what sort options or page I choose, the page source never changes even though the visible content and in-browser source does. Is there any way I can iterate through the pages and parse the different listings or is that beyond my control?

Upvotes: 0

Views: 228

Answers (1)

pguardiario
pguardiario

Reputation: 54984

Here's the url for page 2

http://store.steampowered.com/search/results?sort_by=_ASC&page=2&auction=1&snr=1_7_7_230_7

That should be all you need, parse it the same as page 1.

Upvotes: 1

Related Questions