CODEWITHSUNDEEP
javascriptjsonweb-scrapingcasperjs
Adrien
Adrien

Reputation: 169

Save results from mutiple pages into one JSON file using CasperJS

I am trying to scrape some pages from a website with CasperJS and create a JSON file with the results. However, I am having troubles creating a valid JSON file.

A relevant sample of my code that is run for each page:

casper.then(function(){

var fs = require('fs');

fs.write('results.json', JSON.stringify(result), 'a');
this.echo(JSON.stringify(result, undefined, 4));

casper.then(getNextPage);

});

This creates a JSON file that starts with a "[" and ends with a "]" for each page and looks like this:

[{"Name":"Name1","Price":"10"},(...),{"Name":"Name45","Price":"20"}][{"Name":"Name46","Price":"4.20"}, etc...

I have the following error when I read it in a browser:

"Error: Parse error on line 1:
...key-steam-arma-3/"}][{"Name":"HearthSton
-----------------------^
Expecting 'EOF', '}', ',', ']'

After reading the answers from this similar question I realized that I can't append an object to an existing object because in JSON, there must be one top-level item. However, even if I understand where the error comes from I am not able to find a solution (the solution in this post doesn't work for me since my JSON output is different).

I've been scratching my head for the past few hours trying to come up with a something but with no success. I've tried changing the encoding of the stream to UTF-8 but this doesn't work either. I am sorry if this is a dumb question as I am new to CasperJS and javascript. If anybody could point me to the right direction, this would be really appreciated!

Upvotes: 1

Views: 150

Answers (1)

Andrey E
Andrey E

Reputation: 1

You could create a global variable and append each result to it. Then save it.

var fs = require('fs'); // better to require it once
var results = [];

// ... some code

casper.then(function(){

  results.push(result);

  fs.write('results.json', JSON.stringify(results));
  this.echo(JSON.stringify(results, undefined, 4));

  casper.then(getNextPage);

});

Upvotes: 1

Related Questions