Why doesn't plain JavaScript work in CasperJS?

Question

I'm new to CasperJS. How come this.echo(this.getTitle()); works but console.log("Page Title ", document.title); doesn't? Also why isn't my document.querySelector working? Does anyone have a good explanation? Where in the CasperJS documentation can I find the answer?

Here's my code:

var casper = require('casper').create();
var url = 'http://www.example.com/';

 casper.start(url, function() {
     this.echo(this.getTitle()); // works
     this.echo(this.getCurrentUrl()); // works
});

casper.then(function(){
    this.echo(this.getCurrentUrl()); // works
    console.log("this is URL: ", document.URL); // doesn't work
    console.log("Page Title ", document.title); // doesn't work
    var paragraph = document.querySelectorAll('p')[0].innerHTML;
    console.log(paragraph); // doesn't work
});

casper.run();

EDIT: I'm using casper.thenEvaluate and casper.evaluate now and it's still not working. Any ideas?

var casper = require('casper').create();
var url = 'http://www.example.com/';

casper.start(url, function() {
    this.echo(this.getTitle()); // works
    this.echo(this.getCurrentUrl()); // works
    console.log('page loaded: '); // works
});

casper.thenEvaluate(function(){
    var paragraph = document.querySelectorAll('p')[0].innerHTML; // doesn't work
    console.log(paragraph); // doesn't work
    console.log("Page Title ", document.title); // doesn't work
});

casper.run();

Answer 1

CasperJS has inherited the split between DOM context (page context) and the outer context from PhantomJS. You can only access the sandboxed DOM context through casper.evaluate(). document inside of the evaluate() callback is the variable that you would expect in normal JavaScript, but there is also a document outside of evaluate() which is only a dummy object and doesn't provide access to the DOM of the page.

If you want to access DOM properties, then you need to use evaluate().

var title = casper.evaluate(function(){
    return document.title;
});

But this won't work for DOM nodes, because only primitive objects can be passed out of the DOM context. The PhantomJS documentation says the following:

Note: The arguments and the return value to the evaluate function must be a simple primitive object. The rule of thumb: if it can be serialized via JSON, then it is fine.

Closures, functions, DOM nodes, etc. will not work!

If you want to use document.querySelector(), then you need to produce a representation of a DOM node that can be passed outside:

var form = casper.evaluate(function() {
    var f = document.querySelector('form');
    return { html: f.innerHTML, action: f.action };
});
casper.echo(JSON.stringify(form, undefined, 4));

You can also use all of the available CasperJS functions that can provide representations of DOM nodes such as casper.getElementsInfo().

Also, have a look at Understanding the evaluate function in CasperJS.

Answer 2

Try this.echo(this.fetchText('p')); to get innerhtml. Refer documentation

Answer 3

this.getTitle() executes getTitle() function on Casper object and runs in Casper context, hence it produces the expected result.

However, 'document' is not available in Casper context. The underlying reason is that Casper is running PhantomJS, which is a web browser. So, 'document' is only available in the browser, which is one level "deeper" than the code that runs in Casper context. There is no direct way to share variables between the two environments but there is a way to pass them as parameters by copying the value.

The "bridge" between the two environments (Casper and Phantom) is Casper's 'evaluate' function. Everything inside the function, passed to 'evaluate' as a parameter, will get executed in the browser context, not in Casper context. That's an important distinction. The documentation is available here, as noted by Blender:

http://docs.casperjs.org/en/latest/modules/casper.html#evaluate

Example below:

 casper.evaluate(function(username, password) {
     document.querySelector('#username').value = username;
     document.querySelector('#password').value = password;
     document.querySelector('#submit').click(); 
 }, 'sheldon.cooper', 'b4z1ng4');

In the given example you can see how to pass "username" and "password" parameters from Casper environment to the browser (page) environment.

The anonymous "function(username,password)" will get executed within the browser. Therefore, you can use 'document' inside it.

You can also pass the value back, which can be picked up on Casper side. I.e.

var result = casper.evaluate(function run_in_browser(){
    return document.title;
});

Answer 4

You have to call functions that depend on document with this.evaluate:

var paragraph = this.evaluate(function() {
    return document.querySelector('p').innerHtml;
});

When in doubt, consult the docs.