CODEWITHSUNDEEP
node.jsamazon-web-servicesamazon-s3
natdev
natdev

Reputation: 567

How to list objects using suffix with AWS S3

I am using AWS sdk for javascript and I'm trying to list all objects whose keys end with a specific string (e.g. mp4)

Using listObjects i found an option to specify a prefix, but nothing for a suffix.

Upvotes: 6

Views: 7843

Answers (3)

Mestre San
Mestre San

Reputation: 1915

  1. List the content of the Bucket with your desired prefix
  2. Filter the results and push them into a list that will be returned later
  3. Check if the response from S3 is truncated, which means there are more files
  4. If it is truncated, loop the process passing the NextContinuationToken
  5. Return the accumulated results.

https://gist.github.com/danielsan/935fc05afb155d4c32791017408adb3b

I intentionally added lots of my prefixes to the constants' names because code that you copy from StackOverflow can't be used just anywhere without modification, so you can go ahead and remove all those my prefixes and it will be a very different code for your usage.

Using aws-sdk v2

// https://gist.github.com/danielsan/935fc05afb155d4c32791017408adb3b
import S3 from 'aws-sdk/clients/s3.js'

export const s3 = new S3()

const myFilterContents = (s3Response, pattern) => s3Response.Contents.filter(s3Object => pattern.test(s3Object.Key))

export async function listObjectsWithSuffix (Bucket, Prefix, pattern) {
  let myResults = []
  let myS3Response

  const myS3ListParams = {
    Bucket,
    Prefix,
    MaxKeys: 1000,
    ContinuationToken: undefined
  }

  do {
    myS3Response = await s3.listObjectsV2(myS3ListParams).promise()
    myResults = myResults.concat(myFilterContents(myS3Response, pattern))
    myS3ListParams.ContinuationToken = myS3Response.NextContinuationToken
  } while (myS3Response.IsTruncated === true)

  return myResults
}

using aws-sdk-v3


// npm i @aws-sdk/client-s3
// https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/preview/client/s3/command/ListObjectsV2Command/
import { S3Client, ListObjectsV2Command } from "@aws-sdk/client-s3"

const myS3Client = new S3Client();

const myFilterContents = ({ Contents: c }, pattern) => c.filter(s3Object => pattern.test(s3Object.Key))

export async function listObjectsWithSuffix (Bucket, Prefix, myPattern) {
  let myResults = []
  let myS3Response

  const myS3ListParams = {
    Bucket,
    Prefix,
    MaxKeys: 1000,
    ContinuationToken: undefined
  }

  do {
    myS3Response = await myS3Client.send(new ListObjectsV2Command(myS3ListParams))
    myResults = myResults.concat(myFilterContents(myS3Response, myPattern))
    myS3ListParams.ContinuationToken = myS3Response.NextContinuationToken
  } while (myS3Response.IsTruncated === true)

  return myResults
}

The reason I chose to use a regular expression instead of string.endsWith method is because it gives you much more flexibility.

For example, let's say you have the following file structure in your s3 bucket

index.html
index.js
assets/
a.js
a/
    a1.js
    a2.js
    a3.js
b.js
b/
    b1.js
    b2.js
    b3.js
c.js
c/
    c1.js
    c2.js
    c3.js

The regexp will give you the ability to filter for example only the .js files immediately under the folder assets but not the ones at lower levels, like this

import lib from './your-lib-file.js'

const res = lib.listObjectsWithSuffix('MyBucket', 'assets/', /^assets\/[^/]+\.js/)

Upvotes: 0

Brad
Brad

Reputation: 163262

S3 doesn't support an object key list by suffix. All you can do is fetch a listing of everything and filter in your Node.js application.

Try something like this:

const res = await s3.listObjectsV2({
  Bucket: 'bucket'
}).promise();

const items = res.Contents.filter(item => item.Key.endsWith('mp4'));

Upvotes: 7

John Rotenstein
John Rotenstein

Reputation: 269091

It would be easiest to request all objects, then filter the results within your code.

Upvotes: 0

Related Questions