CODEWITHSUNDEEP
javascriptnode.jscsvstream
unseen_damage
unseen_damage

Reputation: 1376

Pipe NodeJS Stream to an Array

My use case is this: I am looking to read a CSV file in Node and get only the headers. I don't want to write the results of a read stream to a file, rather push the headers to an array once the file is read, so I can take that array and do something to it later on. OR, better yet, take the stream and as it is being read, transform it, then send it to an array. File is a contrived value. I am stuck at this point, where the current output of datafile is an empty array:

const fs = require('fs');
const parse = require('csv-parse');
const file = "my file path";
let dataFile = [];

rs = fs.createReadStream(file);
parser = parse({columns: true}, function(err, data){
    return getHeaders(data)
})

function getHeaders(file){
    return file.map(function(header){
        return dataFile.push(Object.keys(header))
    })
}

What do I need to do in order to get the results I need? I am expecting the headers to be found in an array as the end result.

Upvotes: 1

Views: 8838

Answers (1)

Boris Charpentier
Boris Charpentier

Reputation: 3545

Ok, so there is some confusing things in your code, and one mistake : you didn't actually call your code :)

First, a solution, add this line, after parser : 

rs.pipe(parser).on('end', function(){
    console.log(dataFile);
});

And magic, dataFile is not empty. You stream the file from disk, pass it to the parser, then at the end, call a callback.

For the confusing parts : 

parser = parse({columns: true}, function(err, data){
    // You don't need to return anything from the callback, you give the impression that parser will be the result of getHeaders, it's not, it's a stream.
    return getHeaders(data)
})

function getHeaders(file){
    // change map to each, with no return, map returns an array of the return of the callback, you return an array with the result of each push (wich is the index of the new object).
    return file.map(function(header){
        return dataFile.push(Object.keys(header))
    })
}

And finaly : Please choose with ending line with ; or not, but not a mix ;)

You should end with something like : 

const fs = require('fs');
const parse = require('csv-parse');
const file = "./test.csv";
var dataFile = [];

rs = fs.createReadStream(file);
parser = parse({columns: true}, function(err, data){
    getHeaders(data);
});

rs.pipe(parser).on('end', function(){
    console.log(dataFile);
});

function getHeaders(file){
        file.each(function(header){
            dataFile.push(Object.keys(header));
        });
}

Upvotes: 2

Related Questions