CODEWITHSUNDEEP
javascriptweb-scrapingphantomjs
Umair Ayub
Umair Ayub

Reputation: 21241

Why PhantomJS not scraping the page it is redirected to?

I am scraping http://www.asx.com.au/asx/markets/optionPrices.do?by=underlyingCode&underlyingCode=XJO

It shows a blank white page at first, in that page there is some obfuscated JS code.

That code sends a POST request automatically, and then loads actual page.

I have this code to follow the redirected page, but its not working.

var page;
var myurl = "http://www.asx.com.au/asx/markets/optionPrices.do?by=underlyingCode&underlyingCode=XJO";

var renderPage = function (url) {
    page = require('webpage').create();

    page.onNavigationRequested = function (url, type, willNavigate, main) {
        if (main && url != myurl) {
            myurl = url;
            console.log("redirect caught")

            // GUILTY CODE
            renderPage(url);
        }
    };

    page.open(url, function (status) {
        if (status === "success") {
            console.log("success")
            page.render('yourscreenshot.png');
            phantom.exit(0);
        } else {
            console.log("failed")
            phantom.exit(1);
        }
    });
}

renderPage(myurl);

It only outputs

success
redirect caught

See my code, why GUILTY CODE part is not being executed ... Why renderPage(url) is not being called after redirect caught?

Upvotes: 1

Views: 371

Answers (1)

Jon
Jon

Reputation: 2671

From my understanding phantomJS doesn't really handle redirects well. That may be your issue. You may want to test this in a different way. Or you can use another browser to perform these tests to confirm. Check out this git issue to see what I mean https://github.com/ariya/phantomjs/issues/10389.

Upvotes: 1

Related Questions