CODEWITHSUNDEEP
javascriptcasperjs
ILikeTacos
ILikeTacos

Reputation: 18666

CasperJS sends empty POST data to endpoint

I'm using CasperJS to scrape a site, build a JSON object to organize the info scraped, and then POSTing such JSON object to a service (in PHP) that ETLs the info into a database.

Here's the problem, whenever I try to send the JSON Object the, the object is empty, this is the code I'm using to send the info. 

casper.then(function(){
    var productDetails = {};
    this.each(products,function(self, product){
        self.thenOpen(product, function(a){
            productDetails = this.evaluate(getProductDetails);
        });
        self.thenOpen('http://localhost.endpoint.lan/saveScrapingData.php', {
            method:'post',
            data: {
                name:'Alan',
                info: productDetails
            },
            headers: {
                'Content-type': 'application/x-www-form-urlencoded'
            }
        });
    });
});

the problem is that productDetails is being sent as empty, and I don't know why because if I console.log the variable at this point:

self.thenOpen(product, function(a){
    productDetails = this.evaluate(getProductDetails);
});

the variable is not empty, more than likely this is a racing condition. The variable is being sent before it contains any data, but I don't understand why is this happening, if I'm sending the data after I'm sure the variable contains data.

Upvotes: 0

Views: 660

Answers (1)

Artjom B.
Artjom B.

Reputation: 61892

Yes, at the time of calling thenOpen the productDetails variable is still undefined. You can just split the thenOpen call into then and open so that it is defined when open is actually called.

self.then(function(){
    this.open('http://localhost.endpoint.lan/saveScrapingData.php', {
        method:'post',
        data: {
            name:'Alan',
            info: productDetails
        },
        headers: {
            'Content-type': 'application/x-www-form-urlencoded'
        }
    });
});

This is because of the asynchronous manner of casper steps (then* or wait*). They are scheduled but executed later. The problem is that by calling thenOpen with the settings object fixes the value of productDetails to undefined since the the previous step was not yet executed.

Upvotes: 2

Related Questions