CODEWITHSUNDEEP
javascripthtmlweb-scraping
wwl
wwl

Reputation: 2065

Finding where links are

When I go to this webpage, I see green buttons with the text "信息公开". My task is to download all links of this green button. So if there are ten buttons, I need all ten links.

However I cannot find the text "信息公开" when I download the page in Chrome. I suspect that some Javascript is executed to download information related to "信息公开". Indeed, when I use Chrome to inspect the green buttons, I find information that I cannot find in the HTML files which I download.

How can I find out where the links are?

Upvotes: 0

Views: 67

Answers (1)

Bernd Strehl
Bernd Strehl

Reputation: 2920

You have two JavaScript-based options:

a) Use a headless browser like Phantom.js to scrape the site for the links, there should be no problem with the JavaScript-loaded content. This would be the solution if you want to automate the scraping (like running it daily and posting the links somewhere)

b) Much simpler, but not as automatic: Use the jQuery in the Chrome Console to build a selector to get all the links. For example this piece of code, will give you the links of the yellow community box on the right side of Stack Overflow: $('.community-bulletin a').each(function(){console.log($(this).attr('href'))})

Upvotes: 1

Related Questions