CODEWITHSUNDEEP
javascriptjsonnode.js
user1276919
user1276919

Reputation: 558

What is the best way to include huge json into node.js script?

How to include huge json file (550MB) into node.js script?

I tried:

var json = require('./huge-data-set.json')

The script was executed with increased --max-old-space-size parameter

node --max-old-space-size=4096 diff.js

it failed with:

buffer.js:500
    throw new Error('"toString()" failed');
    ^

Error: "toString()" failed
    at Buffer.toString (buffer.js:500:11)
    at Object.fs.readFileSync (fs.js:552:33)
    at Object.Module._extensions..json (module.js:556:20)

Upvotes: 0

Views: 1309

Answers (2)

Bamieh
Bamieh

Reputation: 10936

Behind Node is v8 engine implementing JS implementation. V8 is written in c++ and as a result it uses types to save its characters in strings (sarcasm used with caution here).

As specified by ECMA JavaScript stores every character on two bytes (utf16 encoded).

The String type is the set of all ordered sequences of zero or more 16-bit unsigned integer values (“elements”) up to a maximum length of 2^53-1 elements

so what you are experiencing is not an out of memory limit, but instead a failed operation since the string is larger than the bytes size of the string type.

if you try to write the json object in javascript instead of reading it from fs (by requiring it), you will have an out of memory exception then, since the limit is set to prevent such cases from happening.

Upvotes: 6

mscdex
mscdex

Reputation: 106736

The error arises because you're trying to create a (single) string that is too large.

Some possible solutions:

  • Use a different backing store (e.g. sqlite db) and query data as you need instead of loading it all into memory at once
  • If your data is an array at the top level, consider having each array element on a separate line, that way you can just buffer a line and JSON.parse() that, then continue to the next line. However, you could still run into memory/GC issues if the parsed JavaScript values eat up too much space.

Upvotes: 3

Related Questions