CODEWITHSUNDEEP
scopeweb-scrapingphantomjscasperjs
Jeff Ryan
Jeff Ryan

Reputation: 585

Using CasperJS with PhantomJS webserver

I have a casper script that submits a form and scrapes the response.

Im trying to set up a "scraping on demand" environment where I can post the form values to a url using PhatomJS webserver, then use that data in my Casper script to scrape the page and then print out the response on the webpage. I don't see how I can pass the post variable into casper and then pass the response back to Phantom.

Heres my basic Phantom/Casper structure:

var server = require('webserver').create();

server.listen(8080, function(request, response) {

phantom.casperPath = '/source/casper/casperjs';
phantom.injectJs('/source/casper/casperjs/bin/bootstrap.js');

var address = request.post.address;

var casper = require('casper').create();

    casper.start();

casper.then(function(){
        address = // want to access global address here
    result = begin(this, address);  //Contians Casper scrape code
});

casper.run(function(){
        this.exit();
});

response.statusCode = 200;
response.write(result);  // from casper
    response.close();
});

Is there any way to access the variables from phantom in casper, and then pass data back once I finish scraping?

Upvotes: 3

Views: 3502

Answers (1)

TJ Nicolaides
TJ Nicolaides

Reputation: 975

Unless you are doing something in PhantomJS that cannot be done in CasperJS, you are probably better off firing up a server in CasperJS and responding with the results of your Casper functions there.

Based on: https://stackoverflow.com/a/16489950/1096083

//includes web server modules
var server = require('webserver').create();


//start web server
var service = server.listen(ip_server, function(request, response) {

    var results;

    var address = request.post.address; // this is not working the way you would expect, needs a little help
    var casper = require('casper').create();

     casper.start(address, function() {
        // do some stuff with casper
        // store it in results
     });

     casper.then(function() {
        // do some more stuff with casper
        // store that in results too
     });

     casper.run(function() {
        response.statusCode = 200;
        response.write(results);
        response.close();              
     });

});

console.log('Server running at http://localhost:' + ip_server+'/');

Upvotes: 4

Related Questions