CODEWITHSUNDEEP
node.jsmongodbgridfs
jsur
jsur

Reputation: 291

MongoDB GridFS API returns corrupt .zip

Context:

Node.js / Loopback application that gets its database populated with a .zip file that contains data from legacy enterprise CRM applications. I'm trying to store a binary of the .zip file into my database for production debugging purposes using GridFS as the file can be >16mb and retrieve it anytime using an admin endpoint.

Problem:

I can store the zip file with function storeLatestZipFile in my module and I can retrieve it using an endpoint that function createLatestZipEndpoint creates.

However, the .zip I'm getting back is larger than the original file (14.7mb vs 21.1mb) and it is also corrupted.

I assume I'm not encoding my data or just not using the GridFS API correctly. Would anybody happen to spot the errors in my code / have more experience in storing .zips with GridFS?

The module in question:

const { pino } = require('amf-logger');
const fs = require('fs');
const mongo = require('mongodb');

const log = pino({ name: 'bot-zip-upload-storage' });

/**
 * @param {string} path Path to the zip file to be persisted.
 * @param {object} app Loopback application instance.
 */
function storeLatestZipFile(path = './', app = {}) {
  log.info('**** Starting streaming current uploaded zip to DB ****');
  const zipReadStream = fs.createReadStream(path, { encoding: 'binary' });
  const { db } = app.dataSources.mongo.connector;
  const bucket = new mongo.GridFSBucket(db);
  bucket.delete('zipfile', () => {
    log.info('deleted old zipfile');

    const uploadStream = bucket.openUploadStreamWithId(
      'zipfile',
      `bot-data-${new Date().toISOString()}`,
      {
        contentType: 'application/zip'
      }
    );

    zipReadStream.pipe(uploadStream);
  });
}

/**
 * @param {object} app Loopback application instance.
 */
async function createLatestZipEndpoint(app = {}) {
  if (!app.get) {
    log.error("app object does not have 'get' property.");
    return;
  }

  app.get('/api/admin/latestzip', async (req, res) => {
    if (!req.headers.latestfile || req.headers.latestfile !== process.env.ADMIN_LATESTFILE) {
      res.sendStatus(403);
      return;
    }

    try {
      const { db } = app.dataSources.mongo.connector;
      const bucket = new mongo.GridFSBucket(db);
      res.writeHead(200, { 'Content-Type': 'application/zip' });
      const downloadStream = bucket.openDownloadStream('zipfile');
      log.info('download stream opened, begin streaming');
      downloadStream.pipe(res);
    } catch (err) {
      log.error(`error getting zipfile: ${err}`);
      res.sendStatus(500);
    }
  });
}

module.exports = {
  storeLatestZipFile,
  createLatestZipEndpoint
};

Upvotes: 2

Views: 756

Answers (1)

Juha Lindstedt
Juha Lindstedt

Reputation: 36

Have you tried createReadStream without buffer-encoding?

const zipReadStream = fs.createReadStream(path);

Upvotes: 2

Related Questions