CODEWITHSUNDEEP
node.jsamazon-s3aws-sdkaws-sdk-nodejs
mike
mike

Reputation: 5223

How to save file from S3 using aws-sdk v3

With the new aws-sdk v3, specifically @aws-sdk/client-s3, it's not clear to me how to store files to disk from S3.
Having the following code

const { Body } = await s3.send(new GetObjectCommand({
  Bucket: 'myBucket',
  Key: 'myKey',
}))

The Body can be of type Readable | ReadableStream | Blob.
I'd like to store this into a file on disk. Since the return types are completely different, is there an easy way to e.g. get a Buffer or Stream.Readable from aws-sdk?

In aws-sdk v2 it was easy with .createReadStream() function which seems to no longer exist.

Upvotes: 21

Views: 19348

Answers (4)

abigail.nguyen
abigail.nguyen

Reputation: 189

To do this in typescript you can also do

import { Readable } from 'stream';
import * as fs from 'fs';

// ... existing s3.send logic ...
await new Promise<void>((resolve, reject) => {
 if (Body instanceof Readable) {
   Body.pipe(fs.createWriteStream(filePath))
    .on('error', err => reject(err))
    .on('close', () => resolve())
 }
});

Upvotes: 6

Oliver Wagner
Oliver Wagner

Reputation: 416

You can use the streaming methods, as mentioned in the responses above, but you can also just use the writeFile method from the fs/promises library:

const { writeFile } = require("node:fs/promises");
const { S3Client, GetObjectCommand } = require("@aws-sdk/client-s3");

const downloadFile = async () => {
    const command = new GetObjectCommand({ Bucket, Key });
    const { Body } = await client.send(command);

    const localFilename = "path_to_where_to_save_file";

    await writeFile(localFilename, Body);
}

and then call this function.

I think this is from node version v14.18.0 onwards, because this method accepts a stream as input.

Upvotes: 6

SRINIVASA YADAV TALAPANTI
SRINIVASA YADAV TALAPANTI

Reputation: 91

GetObjectCommand returns readable stream, we use fs module to create writeStream and pass the returned readableStream from GetObjectCommand to writeStream to save the file 

const fs = require('fs');
const { S3Client, GetObjectCommand } = require('@aws-sdk/client-s3');

const s3Client = new S3Client({
  region: 'your_region',
  credentials: { accessKeyId: 'your_accessKeyId', secretAccessKey: 'your_secretAccessKey' },
});

const bucketParams = {
  Bucket: 'your_bucket',
  Key: 'your_key.jpg',
};

const run = async () => {
  try {
    const data = await s3Client.send(new GetObjectCommand(bucketParams));
    const inputStream = data.Body;
    const downloadPath = 'example.png';
    const outputStream = fs.createWriteStream(downloadPath);
    inputStream.pipe(outputStream);
    outputStream.on('finish', () => {
      console.log(`downloaded the file successfully`);
    });
  } catch (err) {
    console.log('Error', err);
  }
};

Upvotes: 3

mike
mike

Reputation: 5223

It appears ReadableStream | Blob is available only in browser. Node.js always gets Readable which is what I need.

The solution is pretty straightforward afterwards with fs.createWriteStream and piping the Readable such as

await new Promise((resolve, reject) => {
  Body.pipe(fs.createWriteStream(filePath))
    .on('error', err => reject(err))
    .on('close', () => resolve())
})

For reference, here's an issue where improvement to documentation is discussed https://github.com/aws/aws-sdk-js-v3/issues/1877

Upvotes: 21

Related Questions