CODEWITHSUNDEEP
javascriptphantomjs
Curious2learn
Curious2learn

Reputation: 33638

Why does this phantomjs code return null and the document title?

I am trying to learn PhantomJS. I would appreciate if you can help me understand why the code below gives me an error(shown below) and help me fix the error. I am trying to execute some javascript on a page using phantomjs. The code lines in the evaluate function work well when I enter them in Chrome console, i.e., they give the expected result (document.title).

Thank you.

PhantomJS Code

var page = require('webpage').create();

var url = 'http://www.google.com';

page.open(url, function(status) {


    var title = page.evaluate(function(query) {
        document.querySelector('input[name=q]').setAttribute('value', query);

        document.querySelector('input[name="btnK"]').click();

        return document.title;
    }, 'phantomJS');

    console.log(title);

    phantom.exit()
})

Error

TypeError: 'null' is not an object (evaluating 'document.querySelector('input[name="btnK"]').click')

  phantomjs://webpage.evaluate():4
  phantomjs://webpage.evaluate():7
  phantomjs://webpage.evaluate():7
null

Edit 1: In response to Andrew's answer

Andrew, it is strange but on my computer, the button is an input element. The following screenshot shows the result on my computer.

Edit 2: click event unreliable

Sometimes, the following click event works, sometimes it does not.

document.querySelector('input[name="btnK"]')

Not clear to me what is happening.

About the answer

For future readers, in addition to the answer, the gist by Artjom B. is helpful in understanding what is happening. However, for a more robust solution, I think something like the waitfor.js example will have to be used (as suggested in the answer). I hope it is okay to copy and paste Artjom B.'s gist here. While the gist below works (with form submit); it is still not clear to me why it does not work if I try to simulate the click button on the input. If anyone can clarify that, it would be great. 

// Gist by Artjom B.
var page = require('webpage').create();
var url = 'http://www.google.com';
page.open(url, function(status) {
    var query = 'phantomJS';
    page.evaluate(function(query) {
        document.querySelector('input[name=q]').value = query;
        document.querySelector('form[action="/search"]').submit();
    }, query);
    setTimeout(function(){
        var title = page.evaluate(function() {
            return document.title;
        });
        console.log(title);
        phantom.exit();
    }, 2000);
});

Upvotes: 1

Views: 5624

Answers (2)

rknuu
rknuu

Reputation: 329

Google uses a form for submitting its queries. It's also highly likely that google has changed the prototype methods for their search buttons, so it's not really the best site to test web scraping.

The easiest way to do this is to actually perform a form submit, which slightly tweaks your example.

var page = require('webpage').create();
var url = 'http://www.google.com';
page.open(url, function(status) {
    var query = 'phantomJS';
    var title = page.evaluate(function(query) {
        document.querySelector('input[name=q]').value = query;
        document.querySelector('form[action="/search"]').submit();
        return document.title
    }, query);
console.log(title);
phantom.exit();
});

Note that you will likely need to consider that the response is async from this call, so getting the title directly will likely result in an undefined error (you need to account for the time it takes for the page to load before looking up data; you can review this in their waitfor.js example).

Upvotes: 3

Andrew
Andrew

Reputation: 5340

You can open google.com and try document.querySelector('input[name="btnK"]') in the console, it's null. google input

Actully try replace input with button:

document.querySelector('button[name="btnK"]')

google search button

Upvotes: 3

Related Questions