CODEWITHSUNDEEP
javascriptnode.js
Quesofat
Quesofat

Reputation: 1531

Buffer.js Error in Node -- JSON too large

I'm receiving this error 

buffer.js:495
        throw new Error('"toString()" failed');

I'm trying to parse large amounts of data at once using this function: 

exports.markers = function(resp, resp2) {

  this.markersOne = function() {
    var lat, lng = [];
    return resp.map(function(element) {
      lat = element.location_1.coordinates[1];
      lng = element.location_1.coordinates[0];
      return new google.maps.LatLng(lat, lng);
    })
  }

  this.markersTwo = function() {
    return resp2.map(function(element) {
      var lat, lng = [];
      if (element.location_1 != undefined && element.location_1.length > 0) {
        lat = element.location_1.slice(1, -12);
        lng = element.location_1.slice(10, -1);
        return new google.maps.LatLng(lat, lng);
      }
    })
  }

  var result = this.markersOne().concat( this.markersTwo() );
  console.log(result);
  return result;

  }

I have two databases downloaded as JSON files living in the directory.

one is 304 MB and the other is 80MB.

Seeing as how 304 MB is too large, what are other ways I can get around this?

Upvotes: 0

Views: 1216

Answers (1)

matt
matt

Reputation: 1753

Moved my comments to an answer:

Checkout this stackoverflow post. Seems like you have to read in the file with a read stream rather then just require the file.

You could try this library: http://www.github.com/dominictarr/JSONStream

Upvotes: 1

Related Questions