CODEWITHSUNDEEP
arraysjsonnode.jsobjectfor-loop
Octavian Lojnita
Octavian Lojnita

Reputation: 351

JSON File append Objects

I'm trying to iterate over a few dozen links and want to save all results by incrementing each index and fetching the next pageNumber

//dependencies
const jsonfile = require('jsonfile')
const _        = require('lodash')
const utf8     = require('utf8')
const fs       = require('fs')

//config
const file = './data/home.json'
const unique = [ 
                 { suggest: 'København kommune', count: '2577' },
                 { suggest: 'København K, 1000', count: '14' },
                 { suggest: 'København K, 1100', count: '36' } 
               ];

// begin iterations
_.forEach(unique, (data) => {

  var totalPages = Math.ceil(data.count / 99); // get total page numbers for each suggestion

  for ( var i=0; i < totalPages; i++){

    /**
     * Map Cities from our array of unique suggestions
     * Fetch promises for each city
     * @param i {i++} the number of pages we can iterate
     * @param fetch {data.suggest} individual fetch calls
     */

    fetch(`https://home.dk/umbraco/backoffice/home-api/Search?SortType=&SortOrder=&CurrentPageNumber=${i}&SearchResultsPerPage=99&q=${utf8.encode(data.suggest)}&EjendomstypeV1=&EjendomstypeRH=&EjendomstypeEL=&EjendomstypeVL=&EjendomstypeAA=&EjendomstypePL=&EjendomstypeFH=&EjendomstypeLO=&EjendomstypeHG=&EjendomstypeFG=&EjendomstypeNL=&Forretningnr=&ProjectNodeId=&PriceMin=&PriceMax=&EjerudgiftPrMdrMin=&EjerudgiftPrMdrMax=&BoligydelsePrMdrMin=&BoligydelsePrMdrMax=&BoligstoerrelseMin=&BoligstoerrelseMax=&GrundstoerrelseMin=&GrundstoerrelseMax=&VaerelserMin=&VaerelserMax=&Energimaerker%5B%5D=null&ByggaarMin=&ByggaarMax=&EtageMin=&EtageMax=&PlanMin=&PlanMax=&Aabenthus=&Foraeldrekoebsegnet=&Projektsalg=&BoligKanLejes=&MapLngSW=&MapLatSW=&MapLngNE=&MapLatNE=&UserLng=&UserLat=&SearchType=0`)
    .then(response => response.json())
    .then(validate => { 

          validate.searchResults.map(all=> {

                  var result = {
                      longitude:  data.lng, 
                      latitude:   data.lat
                  }


                   jsonfile.writeFileSync(file, result, {spaces: 2, flag: 'a'})
                   //    fs.appendFileSync(file, ',\n'); // my stupid way of adding commas at the end of each object....


          }) // validate.searchResults.map 

    }) // then(validate)
  } // for( i < totalPages )

}); // _.forEach(unique, (data) => {

The problem is that the output does not create trailing commas nor is it enclosed inside an array. I'm using the flag: 'a' method to append objects on each for each loop.... Please help

// bad output
{
  "longitude": 12.5893247949521,
  "latitude": 55.6768760704466
}
{
  "longitude": 12.5821935238857,
  "latitude": 55.6778938800944
}
{
  "longitude": 12.5905159440853,
  "latitude": 55.6803210250784
}

//prefered output

[ // nest all objects inside an array
    {
    "longitude": 12.5893247949521,
    "latitude": 55.6768760704466
    }, // <--- add comma - ,,,,,
    {
    "longitude": 12.5821935238857,
    "latitude": 55.6778938800944
    }, // <--- add comma - ,,,,,
    {
    "longitude": 12.5905159440853,
    "latitude": 55.6803210250784
    }
] // close array

I tried to push everything to one array but it gets to a point where it has like 60.000+ results pushed inside and it kinda freezes the system when it tries to save the results.

I want to append each iteration to the file by using the flag: "a" method or a pipe method... Hope someone has a better way of saving this kind of data to a JSON file!

Thank you very much in advance !

Upvotes: 0

Views: 64

Answers (1)

Felix Kling
Felix Kling

Reputation: 816384

If you want to store an array in the file, then create the array first:

var result = [];
_.forEach(unique, (data) => {
  result.push({
      longitude:  data.lng, 
      latitude:   data.lat
  });
});
jsonfile.writeFileSync(file, result, {spaces: 2, flag: 'a'})

Depending on what unique is, there are more elegant ways. E.g. if it is an array:

var result = unique.map(data => ({
  longitude:  data.lng, 
  latitude:   data.lat
}));

Upvotes: 1

Related Questions