Synchronous execution for node.js program using 'readline'

Question

I'm having some problems getting the asynchronous nature of node to co-operate with me, and after hours of callbacks and googling; I finally turn to you guys.

I have a program that needs to read in lines from a file using the readline module of node. This file contains data that is passed to some asynchronous functions defined within my node program. Once all the data is successfully read and processed, this data needs to be parsed into JSON format, and then outputted.

My problem here is that when I call: readLine.on('close', function() { ...... }, this is run before the asynchronous functions finish running, and therefore I am left with an output of nothing, but the program keeps running the asynchronous functions.

I've created a simple skeleton of functions that should explain my situation more clearly:

function firstAsyncFunc(dataFromFile) {
   //do something asynchronously

   return processedData;
}


function secondAsyncFunc(dataFromFile) {
  //do something else asynchronously

  return processedData;
}


//create readline 
var lineReader = require('readline').createInterface({
  input: require('fs').createReadStream('data.txt')
});


//array to hold all the data processed
var totalDataStorage; 


//read file 
lineReader.on('line', function(line) {

  var processedData = firstAsyncFunction(line);
  var moreProcessedData = secondAsyncFunction(line);


  //store processed data and concatenate into one array
  var tempDataStorage = [{ 'first': processedData, 'second': moreProcessedData }]
  totalDataStorage = totalDataStorage.concat(tempDataStorage);

}).on('close', function() {

  var JSONString = JSON.stringify(... //create JSON for totalDataStorage ...);
  console.log(JSONString); //DOESN'T OUTPUT ANYTHING!

});

I have tried to add a callback to the first/secondAsynFunction, I have tried to make the reading and parsing bit of the program seperate functions, and create callbacks so that parsing is only called when reading finished, but none of those solutions seemed to be working and i'm really struggling - so any help would be appreciated.

Thanks!

EDIT: The data.txt file is of the form

IPData1 DataCenter1
IPData2 DataCenter2
...
IPDataN DataCenterN

I use str.split(" ") to get the respective values, and then pass them appropriately. IPData is a number, and DataCenter is a string

Answer 1

I changed the following and its working locally.

1. use promises to make your life easier.
2. remove .close since you dont have an output defined in the interface.

The 'close' event is emitted when one of the following occur:

1. The rl.close() method is called and the readline.Interface instance has relinquished control over the input and output streams;
2. The input stream receives its 'end' event;
3. The input stream receives -D to signal end-of-transmission (EOT);
4. The input stream receives -C to signal SIGINT and there is no SIGINT event listener registered on the readline.Interface instance.

function firstAsyncFunc(dataFromFile) {
  return new Promise(function(resolve, reject) {
    //do something asynchronously
    resolve(result);
  })
}

function secondAsyncFunc(dataFromFile) {
  return new Promise(function(resolve, reject) {
    //do something asynchronously
    resolve(result);
  })
}

//create readline 
var lineReader = require('readline').createInterface({
  input: require('fs').createReadStream('data.txt')
});

//array to hold all the data processed
var totalDataStorage;

//read file 
lineReader.on('line', function(line) {
  Promise.all([
      firstAsyncFunc(line),
      secondAsyncFunc(line)
    ])
    .then(function(results) {
      var tempDataStorage = [{
        'first': results[0],
        'second': results[1]
      }];
      // i'd use push instead of concat
      totalDataStorage = totalDataStorage.concat(tempDataStorage);
    });
})

Answer 2

Asynchronous functions do not return a value, but you must pass a callback function to it instead. Your line

var processedData = firstAsyncFunction(line);

doesn't make sense at all. If your data.txt file looks like this

IPData1 DataCenter1
IPData2 DataCenter2
IPData3 DataCenter3

you can read data as following

var fs = require('fs');
var rl = require('readline').createInterface({
  input: fs.createReadStream('data.txt')
});
var arr = [];

rl.on('line', a => {
  a = a.split(' ');
  arr.push({
    first: a[0],
    second: a[1]
  });
}).on('close', () => {
  console.log(JSON.stringify(arr, null, 2));
});

It will log

[
  {
    "first": "IPData1",
    "second": "DataCenter1"
  },
  {
    "first": "IPData2",
    "second": "DataCenter2"
  },
  {
    "first": "IPData3",
    "second": "DataCenter3"
  }
]