CODEWITHSUNDEEP
typescriptamazon-web-servicesamazon-s3aws-sdk
ServerMonkey
ServerMonkey

Reputation: 1154

AWS-SDK v3 - Download file with Typescript

Writing a Typescript based Lambda to download a file from a S3 bucket for processing but running into problems with the AWS SDK v3.

I'm getting the error "Property 'pipe' does not exist on type 'Readable | ReadableStream | Blob'. Property 'pipe' does not exist on type 'ReadableStream."

import { S3Client, GetObjectCommand } from "@aws-sdk/client-s3";

const s3Client = new S3Client();        
const command = new GetObjectCommand({ Bucket: myBucket, Key: myFile });

const s3Item = await s3Client.send(command);
s3Item.Body.pipe(createWriteStream(fileName));

Have tried using .pipeTo and .pipeThrough but they report effectively the same error.

Any thoughts as to why this isn't working or how a download can be achieved via v3 of the SDK?

I've had trouble finding download examples, even the official AWS Documentation covers uploading and other steps but not downloading. AWS S3 Examples

Upvotes: 14

Views: 11175

Answers (2)

Eric Haynes
Eric Haynes

Reputation: 5786

The type for Body is a rather clumsy Readable | ReadableStream | Blob | undefined. This is due to the library being usable in both node and the browser, but unfortunately just jams them all into one definition.

  • If using node, it will be Readable
  • In modern browsers, it will be ReadableStream
  • In older browsers, it will be Blob

I have not found a case where Body is undefined, but poking around in the source code, it looks like an artifact of the underlying http layer. So far I've been fine just assuming it's defined, and this makes the signature of helper functions more convenient. However, there may be some cases where that's not safe to assume that I haven't encountered.

Thus, for node, you can use something like this:

import { Readable } from 'stream';
import { GetObjectCommandOutput } from '@aws-sdk/client-s3';

export const asStream = (response: GetObjectCommandOutput) => {
  return response.Body as Readable;
};

export const asBuffer = async (response: GetObjectCommandOutput) => {
  const stream = asStream(response);
  const chunks: Buffer[] = [];
  return new Promise<Buffer>((resolve, reject) => {
    stream.on('data', (chunk) => chunks.push(chunk));
    stream.on('error', (err) => reject(err));
    stream.on('end', () => resolve(Buffer.concat(chunks)));
  });
};

export const asString = async (response: GetObjectCommandOutput) => {
  const buffer = await asBuffer(response);
  return buffer.toString();
};

Or if you want to be stricter about undefined:

const getBody = (response: GetObjectCommandOutput) => {
  return response.Body && (response.Body as Readable);
};

const getBodyAsBuffer = async (response: GetObjectCommandOutput) => {
  const stream = getBody(response);
  if (stream) {
    const chunks: Buffer[] = [];
    return new Promise<Buffer>((resolve, reject) => {
      stream.on('data', (chunk) => chunks.push(chunk));
      stream.on('error', (err) => reject(err));
      stream.on('end', () => resolve(Buffer.concat(chunks)));
    });
  }
};

const getBodyAsString = async (response: GetObjectCommandOutput) => {
  const buffer = await getBodyAsBuffer(response);
  return buffer?.toString();
};

In the browser, the below should work for modern browsers, but definitely be aware of the compatibility here if your app supports IE, as this does not handle the legacy browser Blob responses: https://developer.mozilla.org/en-US/docs/Web/API/Streams_API#browser_compatibility

import { GetObjectCommandOutput } from '@aws-sdk/client-s3';

export const asStream = (response: GetObjectCommandOutput) => {
  return response.Body as ReadableStream;
};

export const asBlob = async (response: GetObjectCommandOutput) => {
  return await new Response(asStream(response)).blob();
};

export const asString = async (response: GetObjectCommandOutput) => {
  return await new Response(asStream(response)).text();
};

I think in the long run, @aws-sdk/lib-storage is going to need a Download similar to its Upload, but haven't heard that it's on any backlog thus far. See this GitHub issue for protracted discussion: https://github.com/aws/aws-sdk-js-v3/issues/1877

Upvotes: 21

ehsk
ehsk

Reputation: 109

Apparently, for node, s3Item.Body is a Readable. In the browser, it is a ReadableStream or Blob (see https://github.com/aws/aws-sdk-js-v3/issues/1877).

import {
  S3Client,
  GetObjectCommand,
  GetObjectCommandOutput
} from '@aws-sdk/client-s3';
import internal from 'stream';

const s3Client = new S3Client();        
const command = new GetObjectCommand({
  Bucket: myBucket,
  Key: myFile
});
const s3Item = await s3Client.send(command);
const commandResult: GetObjectCommandOutput =
  await s3Client.send(new GetObjectCommand(getObjectParams));

if (commandResult.Body instanceof internal.Readable) {
  let readableStream: internal.Readable =
    commandResult.Body as internal.Readable;
  readableStream = readableStream.pipe(...);
} else {
  console.log(`GetObjectCommand should return an
    internal.Readable object. Maybe the code is
    running in the Browser?`);
}

Upvotes: 0

Related Questions