Multer Memory Storage to @google-cloud/storage doesn't complete

Question

I have an Express route that connects to my google-storage bucket and uploads an image. I've tried to stick close to Googles code samples, but go from multer buffer straight to the bucket.

The process appears to complete without returning en err. However, the file appears in the bucket with 0B size. So it seems the meta data arrives but the image buffer does not...??

Can anyone see what I'm doing wrong?

If possible, I'd like to get this to work without the aid of additional NPM help packages like multer-cloud-storage and all the other variants.

const Multer = require("multer");
const multer = Multer({
    storage: Multer.MemoryStorage,
    fileSize: 5 * 1024 * 1024
  });
const {Storage} = require('@google-cloud/storage');
const storage = new Storage({
    projectId: 'project-name',
    keyFile: '../config/project-name.json'
    });

const bucket = storage.bucket('bucket-name');

app.post('/upload', multer.single('file'), function (req, res) {    

    const blob = bucket.file(req.file.originalname);
    const blobStream = blob.createWriteStream({
        metadata: {
            contentType: req.file.mimetype
        },
        resumable: false
    });
    blobStream.on('error', err => {
        next(err);
        console.log(err);
        return;
    });
    blobStream.on('finish', () => {
        blob.makePublic().then(() => {
            res.status(200).send(`Success!`);
        })
    })
    blobStream.end();
})

Answer 1

I was using a similar approach to upload to GCS without third party lib. But, after a wile, a noticed a huge amount of memory usage by node that wasn't garbage collected. Sending 20mb file with Multer.MemoryStorage was resulting in +20MB allocated to my node process and so on.

To fix this, I've made my own Multer storage engine (based on 3rd party packages), to pipe files received direct to GSC stream.

The final code is:

import { Request } from "express";
import { v4 as uuid } from "uuid";
import { Bucket, Storage } from "@google-cloud/storage";
import multer from "multer";

export default class MulterGoogleCloudStorage implements multer.StorageEngine {
  private gcsBucket: Bucket;

  private gcsStorage: Storage;

  private blobFile = {
    filename: ""
  };

  private setBlobFile(req: Request, file: Express.Multer.File) {
    const ext = file.mimetype.split("/")[1].split(";")[0];
    const filename = `${uuid()}_${file.originalname || `.${ext}`}`;

    this.blobFile.filename = filename
      .replace(/^\.+/g, "")
      .replace(/^\/+/g, "")
      .replace(/\r|\n| /g, "_");

    return true;
  }

  constructor() {
    this.gcsStorage = new Storage({
      projectId: process.env.GCLOUD_PROJECT!,
      keyFilename: process.env.GCLOUD_KEYFILE!
    });

    this.gcsBucket = this.gcsStorage.bucket(process.env.GCLOUD_BUCKET!);
  }

  _handleFile = (
    req: Request,
    file: Express.Multer.File,
    cb: (error: Error | null, info?: Partial<Express.Multer.File>) => void
  ): void => {
    if (this.setBlobFile(req, file)) {
      const blobName = this.blobFile.filename;
      const blob = this.gcsBucket.file(blobName);

      const blobStream = blob.createWriteStream();
      file.stream
        .pipe(blobStream)
        .on("error", err => cb(err))
        .on("finish", () => {
          const { name } = blob.metadata;
          cb(null, {
            filename: name,
            size: blob.metadata.size
          });
        });
    }
  };

  _removeFile = (req: Request, file: Express.Multer.File): void => {
    if (this.setBlobFile(req, file)) {
      const blobName = this.blobFile.filename;
      const blob = this.gcsBucket.file(blobName);
      blob.delete();
    }
  };
}

export function storageEngine(): MulterGoogleCloudStorage {
  return new MulterGoogleCloudStorage();
}

Usage:

import multer from "multer";
import MulterGoogleCloudStorage from "../helpers/MulterGoogleCloudStorage";

const upload = multer({
  storage: new MulterGoogleCloudStorage()
});

app.post("/upload", upload.array("medias"), mycontroller.store);

...

Answer 2

You're defining and opening the stream, but not writing anything to it.

If you don'tn want the small overhead of multer-cloud-storage, I'd suggest to look at src/index.ts file. The _handleFile function handles all the writing tasks. The magic happens on

file.stream.pipe(blobStream)
  .on('error', (err) => cb(err))
  .on('finish', (file) => {
    const name = blob.metadata.name;
    const filename = name.substr(name.lastIndexOf('/')+1);
    cb(null, {
        bucket: blob.metadata.bucket,
        destination: this.blobFile.destination,
        filename,
        path: `${this.blobFile.destination}${filename}`,
        contentType: blob.metadata.contentType,
        size: blob.metadata.size,
        uri: `gs://${blob.metadata.bucket}/${this.blobFile.destination}${filename}`,
        linkUrl: `https://storage.cloud.google.com/${blob.metadata.bucket}/${this.blobFile.destination}${filename}`,
        selfLink: blob.metadata.selfLink,
                        })
});

Answer 3

Node's native stream did not work for me. So, streamifier it is.

An Express route that pipes an uploaded image from Multer MemoryStorage to a GCS bucket, bypassing the persisting to disk.

Given this is Googles recommended pattern, I was disappointed at the incompleteness of their code examples and the weak answers here on SO.

    // npm streamifier
    const streamifier = require('streamifier');
    // npm multer
    const Multer = require("multer");
    const multer = Multer({
        storage: Multer.MemoryStorage,
        fileSize: 5 * 1024 * 1024
      });
    // Google Cloud Storage GCS
    const {Storage} = require('@google-cloud/storage');
    const storage = new Storage({
        projectId: 'my-project',
        keyFile: 'my-project.json'
        });
    const bucket = storage.bucket('my-bucket');

    // Express Route Hendler
    app.post('/upload', multer.single('file'), function (req, res) {
        // grab original file name out of multer obj
        const blob = bucket.file(req.file.originalname);
        // create the GCS stream handler
        const blobStream = blob.createWriteStream()
        // Yuck...
        return new Promise((resolve, reject) => {
                streamifier.createReadStream(req.file.buffer)
                    .on('error', (err) => {
                        return reject(err);
                    })
                    .pipe(blobStream)
                    .on('finish', (resp) => {
                       res.send('done') 
                    });
            })
    }); //end route