CODEWITHSUNDEEP
javascriptnode.jsamazon-s3stream
Tamir Nakar
Tamir Nakar

Reputation: 1053

Using AWS SDK (S3.putObject) to upload a Readable stream to S3 (node.js)

My goal is to upload a Readable stream to S3.

The problem is that AWS api seems to accept only a ReadStream as a stream argument.

For instance, the following snippet works just fine:

const readStream = fs.createReadStream("./file.txt") // a ReadStream

await s3.putObject({
    Bucket: this.bucket,
    Key: this.key,
    Body: readStream,
    ACL: "bucket-owner-full-control"
}

Problem starts when I try to do the same with a Readable (ReadStream extends stream.Readable).

The following snippet fails

const { Readable } = require("stream")
const readable = Readable.from("data data data data") // a Readable

await s3.putObject({
    Bucket: this.bucket,
    Key: this.key,
    Body: readable,
    ACL: "bucket-owner-full-control"
}

the error I get from AWS sdk is: NotImplemented: A header you provided implies functionality that is not implemented

*** Note that I prefer a Readable rather than a ReadStream since I'd like to allow passing streams that are not necessarily originated from a file - an in-memory string for instance. So a possible solution could be converting a Readable to a Readstream to work with the SDK.

Any help will be much appreciated!

Upvotes: 2

Views: 6792

Answers (2)

user1689987
user1689987

Reputation: 1546

Example with length calculated and using Buffer:

import { Readable } from "stream";

  saveCreativeImage(name: string, image: Buffer): Promise<string> {
    const options: PutObjectRequest = {
      ACL: 'bucket-owner-full-control',
      Bucket: EnvConfig.S3_CREATIVES_BUCKET_NAME,
      Key: name,
      Body:  Readable.from(image),
      ContentType: 'image/png',
      ContentLength: image.length
    };

Note you used to be able to send Buffer directly but now I think a StreamingBlobTypes is required. Which is defined here: https://github.com/awslabs/smithy-typescript/blob/21ee16a06dddf813374ba88728c68d53c3674ae7/packages/types/src/streaming-payload/streaming-blob-common-types.ts#L22

I think this was changed here: https://github.com/aws/aws-sdk-js-v3/commit/96938415e65ccc4353be8f816e919215de12a1b7#diff-252b2f214c37b3c487fd068bff4968eaa1c8a6085dc9eba0d7bfe15239b05094

Upvotes: 1

Alexander Staroselsky
Alexander Staroselsky

Reputation: 38757

Given the Readable is no longer a file that has metadata such as MIME type and content length you'd need to update your putObject to include those values:

const { Readable } = require("stream")
const readable = Readable.from("data data data data") // a Readable

await s3.putObject({
    Bucket: this.bucket,
    Key: this.key,
    Body: readable,
    ACL: "bucket-owner-full-control",
    ContentType: "text/plain",
    ContentLength: 42 // calculate length of buffer
}

Hopefully that helps!

Upvotes: 4

Related Questions