How can I delete folder on S3 with Node.js?

Question

Yes, I know. There is no folder concept on S3 storage. but I really want to delete a specific folder from S3 with Node.js. I tried two solutions, but both didn't work. My code is below:

Solution 1

Deleting folder directly.

var key='level/folder1/folder2/';
var strReturn;
        var params = {Bucket: MyBucket};
        var s3 = new AWS.S3(params);
        s3.client.listObjects({
            Bucket: MyBucket,
            Key: key
        }, function (err, data) {
            if(err){
                strReturn="{\"status\":\"1\"}";
                
            }else{
                strReturn=+"{\"status\":\"0\"}";
            }
            res.send(returnJson);
            console.log('error:'+err+' data:'+JSON.stringify(data));
        });

Actually, I have a lot of files under folder2. I can delete single file from folder2 if I define key like this: var key='level/folder1/folder2/file1.txt', but it didn't work when I deleted a folder (key='level/folder1/folder2/').

Solution 2

I tried to set expiration to an object when I uploaded this file or folder to s3. code is below:

s3.client.putObject({
                Bucket: Camera_Bucket,
                Key: key,
                            ACL:'public-read', 
                Expires: 60 
            }

But it didn't either. After finishing uploading, I checked the properties of that file. It showed there was nothing value for expiry date:

Expiry Date:none
Expiration Rule:N/A

How can I delete folder on S3 with Node.js?

Answer 1

To recursively delete entire s3 directories and subdirectories.

 module.exports.deleteEntireDirectory = async (dirKey) => {
            
              console.log("deleteEntireDirectory", dirKey);
              console.log('Started recursive delete operation for directory:', dirKey);
            
            
              const bucketName = process.env.BUCKET_NAME;
            
              async function deleteObjects(objects) {
                if (objects.length === 0) return;
            
                console.log(objects);
            
                const deleteParams = {
                  Bucket: bucketName,
                  Delete: { Objects: objects },
                };
            
                await s3.deleteObjects(deleteParams).promise();
              }
            
              async function listAllObjects(prefix) {
                let continuationToken;
                let allObjects = [];
            
                do {
                  const listParams = {
                    Bucket: bucketName,
                    Prefix: prefix,
                    ContinuationToken: continuationToken,
                  };
            
                  const result = await module.exports.clientS3.listObjectsV2(listParams).promise();
                  const contents = result.Contents || [];
                  const prefixes = result.CommonPrefixes || [];
            
                  allObjects = allObjects.concat(contents);
            
                  // Delete objects in the current directory
                  await deleteObjects(contents.map(obj => ({ Key: obj.Key })));
            
                  // Recursively delete subdirectories
                  for (const commonPrefix of prefixes) {
                    console.log(commonPrefix.Prefix);
                    await s3.deleteEntireDirectory(commonPrefix.Prefix);
                  }
            
                  continuationToken = result.NextContinuationToken;
                } while (continuationToken);
            
                return allObjects;
              }
            
              // Start the recursive deletion
              await listAllObjects(dirKey);
              console.log('Recursive delete operation completed for directory:', dirKey);
};

Answer 2

const deleteBucketFolder = async ({ key }) => {
  // this key prop is your folder name
  // don't mistake it for fileName
  const bucketParams = { Bucket: BUCKET, Prefix: key };
  try {
    const listCommand = new ListObjectsCommand(bucketParams);
    const { Contents } = await s3.send(listCommand);
    for (let index = 0; index < Contents.length; index++) {
      const element = Contents[index];
      await s3.send(
        new DeleteObjectCommand({ Bucket: BUCKET, Key: element.Key })
      );
    }
    return { message: "Success. Folder deleted." };
  } catch (err) {
    throw Error;
  }
};

Answer 3

According to accepted answer I created promise returned function, so you can chain it.

function emptyBucket(bucketName){
    let currentData;
    let params = {
        Bucket: bucketName,
        Prefix: 'folder/'
    };

    return S3.listObjects(params).promise().then(data => {
        if (data.Contents.length === 0) {
            throw new Error('List of objects empty.');
        }

        currentData = data;

        params = {Bucket: bucketName};
        params.Delete = {Objects:[]};

        currentData.Contents.forEach(content => {
            params.Delete.Objects.push({Key: content.Key});
        });

        return S3.deleteObjects(params).promise();
    }).then(() => {
        if (currentData.Contents.length === 1000) {
            return emptyBucket(bucketName);
        } else {
            return true;
        }
    });
}

Answer 4

Here is a do-while typescript implementation using @aws-sdk/client-s3 v3:

import {DeleteObjectsCommand, ListObjectsV2Command, ListObjectsV2CommandOutput, S3Client} from '@aws-sdk/client-s3';

const s3client = new S3Client({...});

async function emptyBucketByPrefix(bucket: string, prefix: string) {
    let listResponse: ListObjectsV2CommandOutput | undefined;
    do {
        listResponse = await s3client.send(new ListObjectsV2Command({Bucket: bucket, Prefix: prefix}));
        if (!listResponse.Contents?.length) {
            break;
        }
        const objects = listResponse.Contents.map(({Key}) => ({Key}));
        const command = new DeleteObjectsCommand({
            Bucket: bucket,
            Delete: {
                Objects: objects,
            },
        });
        await s3client.send(command);
    } while (listResponse.IsTruncated);
}

Answer 5

A much simpler way is to fetch all objects (keys) at that path & delete them. In each call fetch 1000 keys & s3 deleteObjects can delete 1000 keys in each request too. Do that recursively to achieve the goal

Written in typescript

/**
 * delete a folder recursively
 * @param bucket
 * @param path - without end /
 */
deleteFolder(bucket: string, path: string) {
    return new Promise((resolve, reject) => {
        // get all keys and delete objects
        const getAndDelete = (ct: string = null) => {
            this.s3
                .listObjectsV2({
                    Bucket: bucket,
                    MaxKeys: 1000,
                    ContinuationToken: ct,
                    Prefix: path + "/",
                    Delimiter: "",
                })
                .promise()
                .then(async (data) => {
                    // params for delete operation
                    let params = {
                        Bucket: bucket,
                        Delete: { Objects: [] },
                    };
                    // add keys to Delete Object
                    data.Contents.forEach((content) => {
                        params.Delete.Objects.push({ Key: content.Key });
                    });
                    // delete all keys
                    await this.s3.deleteObjects(params).promise();
                    // check if ct is present
                    if (data.NextContinuationToken) getAndDelete(data.NextContinuationToken);
                    else resolve(true);
                })
                .catch((err) => reject(err));
        };

        // init call
        getAndDelete();
    });
}

According doc at https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html:

A response can contain CommonPrefixes only if you specify a delimiter.

CommonPrefixes contains all (if there are any) keys between Prefix and the next occurrence of the string specified by the delimiter.

Omitting Delimiter parameter will make ListObject return all keys starting by the Prefix parameter.

Answer 6

listObjectsV2 list files only with current dir Prefix not with subfolder Prefix. If you want to delete folder with subfolders recursively this is the source code: https://github.com/tagspaces/tagspaces-common/blob/develop/packages/common-aws/io-objectstore.js#L1060

  deleteDirectoryPromise = async (path: string): Promise<Object> => {
    const prefixes = await this.getDirectoryPrefixes(path);

    if (prefixes.length > 0) {
      const deleteParams = {
        Bucket: this.config.bucketName,
        Delete: { Objects: prefixes }
      };

      return this.objectStore.deleteObjects(deleteParams).promise();
    }
    return this.objectStore
      .deleteObject({
        Bucket: this.config.bucketName,
        Key: path
      })
      .promise();
  };

  /**
   * get recursively all aws directory prefixes
   * @param path
   */
  getDirectoryPrefixes = async (path: string): Promise<any[]> => {
    const prefixes = [];
    const promises = [];
    const listParams = {
      Bucket: this.config.bucketName,
      Prefix: path,
      Delimiter: '/'
    };
    const listedObjects = await this.objectStore
      .listObjectsV2(listParams)
      .promise();

    if (
      listedObjects.Contents.length > 0 ||
      listedObjects.CommonPrefixes.length > 0
    ) {
      listedObjects.Contents.forEach(({ Key }) => {
        prefixes.push({ Key });
      });

      listedObjects.CommonPrefixes.forEach(({ Prefix }) => {
        prefixes.push({ Key: Prefix });
        promises.push(this.getDirectoryPrefixes(Prefix));
      });
      // if (listedObjects.IsTruncated) await this.deleteDirectoryPromise(path);
    }
    const subPrefixes = await Promise.all(promises);
    subPrefixes.map(arrPrefixes => {
      arrPrefixes.map(prefix => {
        prefixes.push(prefix);
      });
    });
    return prefixes;
  };

Answer 7

I suggest you to do it in 2 steps, so you can "follow" whats happen (with a progressBar etc...):

1. Get all keys to remove
2. Remove keys

Of course , the #1 is a recursive function, such as:

https://gist.github.com/ebuildy/7ac807fd017452dfaf3b9c9b10ff3b52#file-my-s3-client-ts

import { ListObjectsV2Command, S3Client, S3ClientConfig } from "@aws-sdk/client-s3"


/**
   * Get all keys recurively
   * @param Prefix 
   * @returns 
   */
  public async listObjectsRecursive(Prefix: string, ContinuationToken?: string): Promise<
    any[]
  > {
    // Get objects for current prefix
    const listObjects = await this.client.send(
      new ListObjectsV2Command({
        Delimiter: "/",
        Bucket: this.bucket.name,
        Prefix,
        ContinuationToken
      })
    );

    let deepFiles, nextFiles

    // Recurive call to get sub prefixes
    if (listObjects.CommonPrefixes) {
      const deepFilesPromises = listObjects.CommonPrefixes.flatMap(({Prefix}) => {
        return this.listObjectsRecursive(Prefix)
      })

      deepFiles = (await Promise.all(deepFilesPromises)).flatMap(t => t)
    }

    // If we must paginate
    if (listObjects.IsTruncated) {
      nextFiles = await this.listObjectsRecursive(Prefix, listObjects.NextContinuationToken)
    }

    return [
      ...(listObjects.Contents || []),
      ...(deepFiles || []),
      ...(nextFiles || [])
    ]
  }

Then, delete all objects:

  public async deleteKeys(keys: string[]): Promise<any[]> {
    
    function spliceIntoChunks(arr: any[], chunkSize: number) {
      const res = [];
      while (arr.length > 0) {
          const chunk = arr.splice(0, chunkSize);
          res.push(chunk);
      }
      return res;
    }

    const allKeysToRemovePromises =  keys.map(k => this.listObjectsRecursive(k))
    const allKeysToRemove = (await Promise.all(allKeysToRemovePromises)).flatMap(k => k)      
    const allKeysToRemoveGroups = spliceIntoChunks(allKeysToRemove, 3)

    const deletePromises = allKeysToRemoveGroups.map(group => {
      return this.client.send(
        new DeleteObjectsCommand({
          Bucket: this.bucket.name,
          Delete: {
            Objects: group.map(({Key}) => {
              return {
                Key
              }
            })
          }
        })
      )
    })

    const results = await Promise.all(deletePromises)

    return results.flatMap(({$metadata, Deleted}) => {
      return Deleted.map(({Key}) => {
        return {
          status: $metadata.httpStatusCode,
          key: Key
        }
      })
    })
  }

Answer 8

Better solution with @aws-sdk/client-s3 module:

private async _deleteFolder(key: string, bucketName: string): Promise<void> {
  const DeletePromises: Promise<DeleteObjectCommandOutput>[] = [];
  const { Contents } = await this.client.send(
    new ListObjectsCommand({
      Bucket: bucketName,
      Prefix: key,
    }),
  );
  if (!Contents) return;
  Contents.forEach(({ Key }) => {
    DeletePromises.push(
      this.client.send(
        new DeleteObjectCommand({
          Bucket: bucketName,
          Key,
        }),
      ),
    );
  });

  await Promise.all(DeletePromises);
}

ListObjectsCommand returns the keys of files in the folder, even with subfolders

Answer 9

The accepted answer throws an error when used in typescript. I made it work by modifying the code in the following way. I'm very new to Typescript but at least it is working now.

 async function emptyS3Directory(prefix: string) {
  const listParams = {
    Bucket: "bucketName",
    Prefix: prefix, // ex. path/to/folder
  };

  const listedObjects = await s3.listObjectsV2(listParams).promise();

  if (listedObjects.Contents.length === 0) return;

  const deleteParams = {
    Bucket: bucketName,
    Delete: { Objects: [] as any },
  };

  listedObjects.Contents.forEach((content: any) => {
    deleteParams.Delete.Objects.push({ Key: content.Key });
  });

  await s3.deleteObjects(deleteParams).promise();

  if (listedObjects.IsTruncated) await emptyS3Directory(prefix);
}

Answer 10

I like the list objects and then delete approach, which is what the aws cmd line does behind the scenes btw. But I didn't want to await the list (few seconds) before deleting them. So I use this 1 step (background) process, I found it slightly faster. You can await the child process if you really want to confirm deletion, but I found that took around 10 seconds, so I don't bother I just fire and forget and check logs instead. The entire API call with other stuff now takes 1.5s which is fine for my situation.

var CHILD = require("child_process").exec;
function removeImagesAndTheFolder(folder_name_str, callback){
            
            var cmd_str = "aws s3 rm s3://" 
                    + IMAGE_BUCKET_STR 
                    + "/" + folder_name_str
                    + "/ --recursive";
    
            if(process.env.NODE_ENV === "development"){
                //When not on an EC2 with a role I use my profile    
                cmd_str += " " + "--profile " + LOCAL_CONFIG.PROFILE_STR;
            }
            // In my situation I return early for the user. You could make them wait tho'.
            callback(null, {"msg_str": "Check later that these images were actually removed."});
            //do not return yet still stuff to do   
            CHILD(cmd_str, function(error, stdout, stderr){
                if(error || stderr){
                    console.log("Problem removing this folder with a child process:" + stderr);
                }else{
                    console.log("Child process completed, here are the results", stdout);
                }
            });
        }

Answer 11

You can use aws-sdk module for deleting folder. Because you can only delete a folder when it is empty, you should first delete the files in it. I'm doing it like this :

function emptyBucket(bucketName,callback){
  var params = {
    Bucket: bucketName,
    Prefix: 'folder/'
  };

  s3.listObjects(params, function(err, data) {
    if (err) return callback(err);

    if (data.Contents.length == 0) callback();

    params = {Bucket: bucketName};
    params.Delete = {Objects:[]};
    
    data.Contents.forEach(function(content) {
      params.Delete.Objects.push({Key: content.Key});
    });

    s3.deleteObjects(params, function(err, data) {
      if (err) return callback(err);
      if (data.IsTruncated) {
        emptyBucket(bucketName, callback);
      } else {
        callback();
      }
    });
  });
}

Answer 12

You can try this:

import { s3DeleteDir } from '@zvs001/s3-utils'
import { S3 } from 'aws-sdk'

const s3Client = new S3() 

await s3DeleteDir(s3Client, {
  Bucket: 'my-bucket',
  Prefix: `folder/`,
})

Answer 13

You can delete an empty folder the same way you delete a file. In order to delete a non-empty folder on AWS S3, you'll need to empty it first by deleting all files and folders inside. Once the folder is empty, you can delete it as a regular file. The same applies to the bucket deletion. We've implemented it in this app called Commandeer so you can do it from a GUI.

Answer 14

According to Emi's answer I made a npm package so you don' t need to write the code yourself. Also the code is written in typescript.

See https://github.com/bingtimren/s3-commons/blob/master/src/lib/deleteRecursive.ts

Answer 15

Here is an implementation in ES7 with an async function and using listObjectsV2 (the revised List Objects API):

async function emptyS3Directory(bucket, dir) {
    const listParams = {
        Bucket: bucket,
        Prefix: dir
    };

    const listedObjects = await s3.listObjectsV2(listParams).promise();

    if (listedObjects.Contents.length === 0) return;

    const deleteParams = {
        Bucket: bucket,
        Delete: { Objects: [] }
    };

    listedObjects.Contents.forEach(({ Key }) => {
        deleteParams.Delete.Objects.push({ Key });
    });

    await s3.deleteObjects(deleteParams).promise();

    if (listedObjects.IsTruncated) await emptyS3Directory(bucket, dir);
}

To call it:

await emptyS3Directory(process.env.S3_BUCKET, 'images/')