CODEWITHSUNDEEP
javascriptarraysjsonnode.js
Stas Coder
Stas Coder

Reputation: 347

Process large json array in node.js

I use axios to get data from API then consume data in my node.js app. The data are array of 300 objects like this one:

{
  'location': 'us',
  'date': '156565665',
  'month': '5.1',
  'day': '6',
  'type': 'default',
  'count': '7',
  'title': 'good',
  'filter': 'no',
  'duration': 'short',
  'options': 'no',
}

After I get this array of objects I need to transform each object: replace its keys with new ones and convert some values into proper data types (string to float):

{
  'loc': 'us',
  'date_new': parseInt('156565665'),
  'month': parseFloat('5.1'),
  'day': parseInt('6'),
  'type': 'default',
  'count': parseInt('7'),
  'title': 'good',
  'filter': 'no',
  'duration': 'short',
  'options': 'no',
}

For now I just use for loop and in each iteration convert keys and values of each object. But there will be thousands of objects like these ones. It will be a worker for processing these data. What is the best way to process them in node.js?

I am going to use some ready-made queue like bee-queue or resque, but even in this case it would be good to make code "node.js way" that this processing of my array of objects will not slow down node loop. Maybe use push each object to array of promises and put them to Promise.all() (but there will be 300 promises in Promise.all())? What is the best way to make hard calculations like this in node.js?

Upvotes: 3

Views: 4775

Answers (1)

gurvinder372
gurvinder372

Reputation: 68443

But there will be thousands of objects like these ones. It will be a worker for processing these data. What is the best way to process them in node.js?

I would recommend

Example

var request = require('request')
  , JSONStream = require('JSONStream')
  , es = require('event-stream')

request({url: 'URL'})
  .pipe(JSONStream.parse('rows.*'))
  .pipe(es.mapSync(function (data) {
    console.error(data)
    return data
  }))

  • After parsing, store them in a database instead of processing them immediately since a hard-calculation for a big object will hold-up the processing on Nodejs.

  • Pick them up individually one by one from database for processing.

Upvotes: 3

Related Questions