CODEWITHSUNDEEP
ajaxweb-scraping
user143410
user143410

Reputation: 63

Finding URL loaded via AJAX

In this answer to another question, a user was able to locate the URL for a part of this page which is loaded via javascript. So far, I have been unable to duplicate this simple feat: In particular, I have looked at the page source, but have been unable to locate the code to which the responder refers:

<script>populator = new Populator({parentId:
"profileForm:vanguardFundTabBox:tab0",execOnLoad:true,
 populatorUrl:"/us/JSP/Funds/VGITab/VGIFundOverviewTabContent.jsf?FundIntExt=INT&FundId=0542",
inline:fals   e,type:"once"});
</script>

I have searched the page source for JS which is written locally on the page's HTML file and also within JS files referenced in the source file. What have I overlooked?

Upvotes: 1

Views: 79

Answers (1)

Allan Cameron
Allan Cameron

Reputation: 173793

TL;DR the Q&A referred to is over 10 years old, and the website has changed since then.

There are a couple of things you may be missing. Firstly, the Stack Overflow question you are referring to is over 10 years old. The page it refers to strictly doesn't exist anymore. If you use Chrome or Firefox developer tools, you will notice that requesting the page link gives you an http 301, indicating the page has moved permanently, and directs you to a different page which presumably looks similar to the page from 10 years ago. The current version of the page is built using angular js, which was not in widespread use in 2009 (in fact, I think it was only created in 2009). The reason that you can't find the ajax request anywhere is that this is no longer how the page is constructed.

You could probably still replicate the feat of scraping the page, but now it would be much harder. You would have to do it by requesting and parsing the JSON that is now used to populate the page's fields, e.g. from https://api.vanguard.com/rs/ire/01/ind/fund/VWIUX/expense.jsonp?callback=angular.callbacks._w&planId=null . However, this will only work if you have the correct cookies in the header, etc.

So the good news is, you aren't failing to follow the method given in the answer. The bad news is that the answer is obsolete.

Upvotes: 1

Related Questions