CODEWITHSUNDEEP
mongodbcsvmongoose
Shrikant
Shrikant

Reputation: 334

Mongoose stream data and export data to CSV file

I'm trying to export a CSV file with following codes but its not working. I'm assuming that my data will be huge, hence Mongoose cursor will be the option. I also notice that I had mention 'batchSize' of 500 still it spitting 1 at a time.

module.exports.export2CSV = (req, res)=>{
  var limit       = 250000;
  var filename    = 'export.csv';
  var query       = User.find();
  if (limit){query.limit(limit);} 
  var headers = {
      'Content-Type': 'text/csv',
      'Content-disposition': 'attachment;filename=' + filename
  }
  // res.writeHead(200, headers)
  res.setHeader('Content-disposition', 'attachment; filename='+filename);
  res.set('Content-Type', 'text/csv');


  var stream = User.aggregate().cursor({ batchSize: 500 }).exec().stream();
  stream.on('data', function (doc) {
    // do whatever you want with the data
  res.write(doc);
}).on('error', function (err) {
  // handle errors
  res.end();
}).on('end', function () {
  // close files
  res.end();
});
}

Right now I'm calling this code directly but later I will be calling it from a Ajax request.

Please suggest the best and fastest way of doing it.

Upvotes: 2

Views: 8582

Answers (2)

May Hemade
May Hemade

Reputation: 21

With mongoose 6+ and fast-csv 3.x, here is an updated solution.

import * as fastCsv from "fast-csv"

const User = mongoose.model('Users', UserSchema);

  const cursor = User.find().cursor();

  const transformer = (doc)=> {
    return {
        Id: doc._id,
        Name: doc.fullname,
        Email: doc.email,
        Type: doc.registration_type,
        RegisterOn: doc.registered_on
    };
  }

  const filename = 'export.csv';

  res.setHeader('Content-disposition', `attachment; filename=${filename}`);
  res.writeHead(200, { 'Content-Type': 'text/csv' });

  res.flushHeaders();

  const csvStream = fastCsv.format({headers: true}).transform(transformer)
  cursor.pipe(csvStream).pipe(res);

Upvotes: 2

Shrikant
Shrikant

Reputation: 334

Here the code which solved my problem

  var User = mongoose.model('Users', UserSchema);

  const cursor = User.find();

  const transformer = (doc)=> {
    return {
        Id: doc._id,
        Name: doc.fullname,
        Email: doc.email,
        Type: doc.registration_type,
        RegisterOn: doc.registered_on
    };
  }

  const filename = 'export.csv';

  res.setHeader('Content-disposition', `attachment; filename=${filename}`);
  res.writeHead(200, { 'Content-Type': 'text/csv' });

  res.flushHeaders();

  var csvStream = fastCsv.createWriteStream({headers: true}).transform(transformer)
  cursor.stream().pipe(csvStream).pipe(res);

Upvotes: 7

Related Questions