CODEWITHSUNDEEP
javascriptpaginationweb-scrapingphantomjscasperjs
dsp_099
dsp_099

Reputation: 6121

Dynamic number of CasperJS steps for paginating and web scraping inside a popup

Trying to scrape a JS-heavy, paginated list.

I can see the number of total pages and so therefore I know how many times I will need to run .waitForSelectorTextChange(). This doesn't work:

casper.withPopup(/NewInventoryManager/, function() {
  var target = 3;
  for (var i = 0; i < target; i++) {
    this.click('#btnNext'); // Click next and initiate a loading.gif modal
    casper.waitForSelectorTextChange('#some_div', function() {
      this.echo('Here I can parse the now updated DOM.');
    }, function() {}, 35000);
  }
});

It runs through the script all at once and sets the .waitForSelectorTextChange functions all at once, when the initial click causes the DOM to change.

How can I add a dynamic number of steps inside the script programmatically? It seems to be working, sort of. How do I make them go one after the other?

Upvotes: 1

Views: 789

Answers (1)

dsp_099
dsp_099

Reputation: 6121

Inspired by this answer, here's how the recursion would look inside a .withPopup (the same really):

function scrapeAndClick() {
  // Total pages count
  target = this.evaluate(function addID() { return $('#pagecountspan').text(); });
  target = parseInt(target.replace(/[^0-9.]/g, ''));

  // Current page (usually 1)
  var current = casper.evaluate(function getCurrent() { return parseInt($('#ddlpaging').val()); });

  casper.waitForSelectorTextChange('#moduleInventoryBrowserWrapper', function() {
    this.echo('The text on #moduleInventoryBrowserWrapper has been changed :: Scrape now!');

    // Now + 1
    current = casper.evaluate(function getCurrent() { return parseInt($('#ddlpaging').val()); });

    if (current < target) {
      casper.then(scrapeAndClick);
    } else {
      casper.echo('Finished!');
    }

  }, function onTm() { this.echo('timeout 1'); }, 35000);


}

casper.withPopup(/NewInventoryManager/, function() {
  this.echo('Popup detected: ' + this.getTitle());

  casper.then(scrapeAndClick);

});

Upvotes: 1

Related Questions