CODEWITHSUNDEEP
node.jsamazon-web-servicesexpressamazon-s3multer
Richard Robinson
Richard Robinson

Reputation: 997

AWS S3 Node.js SDK `NotImplemented` Error with Multer

I'm trying to set up an endpoint to upload files to an AWS S3 bucket. I am using Node.JS, Express, the AWS S3 SDK, and the Multer library.

This is the code I'm using currently for the uploading middleware:

export const uploadMiddleware = multer({
  storage: {
    _handleFile(
      req: Request,
      file: Express.Multer.File,
      callback: (error?: any, info?: Partial<Express.Multer.File>) => void
    ) {
      const key = crypto.randomBytes(16).toString('hex');
      s3.send(
        new PutObjectCommand({
          Bucket: 'my-bucket',
          Body: file.stream,
          Key: key,
          ContentLength: file.size,
          ContentType: file.mimetype,
        }),
        (err, output) => {
          if (err) {
            console.log(err);
            callback(err, undefined);
          } else {
            console.log(output);
            callback(null, file);
          }
        }
      );
    },
    _removeFile(
      req: Request,
      file: Express.Multer.File,
      callback: (error: Error) => void
    ) {
      // @ts-ignore
      callback(null);
    },
  },
}).array('files');

However, the S3 SDK is giving this error:

NotImplemented: A header you provided implies functionality that is not implemented
2021-07-11T00:18:48.557532998Z     at deserializeAws_restXmlPutObjectCommandError (/usr/src/app/node_modules/@aws-sdk/client-s3/protocols/Aws_restXml.ts:9515:39)
2021-07-11T00:18:48.557542041Z     at processTicksAndRejections (internal/process/task_queues.js:93:5)
2021-07-11T00:18:48.557545571Z     at /usr/src/app/node_modules/@aws-sdk/middleware-serde/src/deserializerMiddleware.ts:18:20
2021-07-11T00:18:48.557548319Z     at /usr/src/app/node_modules/@aws-sdk/middleware-signing/src/middleware.ts:26:22
2021-07-11T00:18:48.557551272Z     at StandardRetryStrategy.retry (/usr/src/app/node_modules/@aws-sdk/middleware-retry/src/StandardRetryStrategy.ts:83:38)
2021-07-11T00:18:48.557554272Z     at /usr/src/app/node_modules/@aws-sdk/middleware-logger/src/loggerMiddleware.ts:22:22 {
2021-07-11T00:18:48.557557035Z   Code: 'NotImplemented',
2021-07-11T00:18:48.557559463Z   Header: 'Transfer-Encoding',
2021-07-11T00:18:48.557561825Z   RequestId: 'T6QNENNWC0X2WDXP',
2021-07-11T00:18:48.557564202Z   HostId: 'gJo7gOiFu9RjdvishH0t6EP8/BHDlBW913gicSlYh1eyJ/JkZaX9QSepRmdeq5Gxt4lg6aKMxxI=',
2021-07-11T00:18:48.557566724Z   '$fault': 'client',
2021-07-11T00:18:48.557569116Z   '$metadata': {
2021-07-11T00:18:48.557571528Z     httpStatusCode: 501,
2021-07-11T00:18:48.557573948Z     requestId: undefined,
2021-07-11T00:18:48.557576323Z     extendedRequestId: 'gJo7gOiFu9RjdvishH0t6EP8/BHDlBW913gicSlYh1eyJ/JkZaX9QSepRmdeq5Gxt4lg6aKMxxI=',
2021-07-11T00:18:48.557578821Z     cfId: undefined,
2021-07-11T00:18:48.557589903Z     attempts: 1,
2021-07-11T00:18:48.557592818Z     totalRetryDelay: 0
2021-07-11T00:18:48.557595524Z   },
2021-07-11T00:18:48.557597888Z   storageErrors: []
2021-07-11T00:18:48.557600254Z }

I have tried multiple possibly solutions, such as including/excluding ContentLength and ContentType, and using file.buffer instead of file.stream. I've also tried other solutions from related StackOverflow posts, but nothing has worked.

Note that I am also aware that NPM packages for Multer/S3 interoperability exist, however none use the new modular AWS system and so I opted to use my own.

Any help would be appreciated, thanks!

Upvotes: 6

Views: 11979

Answers (2)

Andriy Kharchuk
Andriy Kharchuk

Reputation: 1304

The PutObject S3 API requires a known stream length. So, if the 'content-length' header is missing, the API call will fail.

The alternative solution is using @aws-sdk/lib-storage library:

import { Stream } from 'stream';
import { S3 } from '@aws-sdk/client-s3';
import { Upload } from "@aws-sdk/lib-storage";
// skipped code

    const passThrough = new Stream.PassThrough();

    file.stream.pipe(passThrough);

    const upload = new Upload({
        client: new S3({}),
        params: {
            Bucket: 'my-bucket',
            Key: key,
            Body: passThrough,
            ContentType: file.mimetype,
        }
    });

    upload.on('s3 upload progress', (p) => console.log(p));

    await upload.done();

Upvotes: 14

Richard Robinson
Richard Robinson

Reputation: 997

I had to convert the stream to a buffer in the handleFile function:

async function streamToBuffer(stream: Stream): Promise<Buffer> {
  return new Promise<Buffer>((resolve, reject) => {
    const _buf: any[] = [];

    stream.on('data', (chunk) => _buf.push(chunk));
    stream.on('end', () => resolve(Buffer.concat(_buf)));
    stream.on('error', (err) => reject(err));
  });
}

export function handleFile(
  req: Request,
  file: Express.Multer.File,
  callback: (error?: any, info?: Express.Multer.File) => void
) {
  const key = generateId();

  streamToBuffer(file.stream)
    .then((buffer) =>
      s3.send(
        new PutObjectCommand({
          Bucket: config.aws.issueNewBucket,
          Body: buffer,
          Key: key,
          ACL: 'public-read',
          ContentType: file.mimetype,
        })
      )
    )
    .then(() => callback(null, file))
    .catch((err) => callback(err));
}

Upvotes: 10

Related Questions