CODEWITHSUNDEEP
javaphpajaxcurlscreen-scraping
Coldblackice
Coldblackice

Reputation: 3520

Is there a library than can trudge through AJAX/javascript?

I'm using PHP to scrape some information off webpages, however, I've discovered that the info I'm trying to scrape from the pages is loading through some manner of AJAX/javascript. I thought I remembered that Curl could iterate through the javascript, but I've found that that's not the case.

I seem to remember some sort of backend "web browser" library/function that could trace through javascript and AJAX, to get at a final page result of what a full-functioned browser would arrive at.

Is there a library or function that can do this? Any ideas on how to go about this, other than having to manually trace through the scripts/redirects myself? It doesn't have to be pretty -- I'm just looking to scrape the resulting text.

Upvotes: 0

Views: 97

Answers (2)

pguardiario
pguardiario

Reputation: 54984

Maybe not in php but in other languages there's: Watir/WatiN, selenium, watir/selenium-webdriver, capybara-webkit, celerity, node.js runs js directly, as well as phantomjs. There's also iMacros and similar commercial options.

But I usually find that I can get the data I want without any of these by just looking at the requests the page is making and recreate them/parsing the response.

Upvotes: 1

Aleks G
Aleks G

Reputation: 57316

I don't think there is such a library. If you're really desperate and you have lots of time on your hands, then you can, of course, download source code of Firefox, for example, and build yourself something useful. However I don't think this is going to be the best use of yours or anybody else's resources.

Note that even google's indexing bot does not process ajax. Here is what Google has to say about it. It's quite possible that the site you're dealing with does support this, in which case you can try using this google's technique, but on the whole, unfortunately, you're out of luck.

Upvotes: 1

Related Questions