CODEWITHSUNDEEP
amazon-web-servicesamazon-s3aws-sdk
romain-nio
romain-nio

Reputation: 1205

Fatest way to retrieve S3 objects metadata (NodeJS API)

I stored custom metadatas in my S3 objects (I stored a descriprition of the file in the meta x-amz-metadata-description).

I'd like to retrieve a file list with S3 objects properties, including metadatas.

I thought that s3.listObjectsV2 can help me but it seems that this method cannot retrieve meta data (see official doc here)

I saw that I can retrieve those medata with getObjects, but it will increase a lot the number of call and the latency of the request.

Do you have any idea to bulk retrieve metadatas of a list of objects ?

Thanks, Romain.

Upvotes: 1

Views: 5244

Answers (2)

Dilip Kola
Dilip Kola

Reputation: 212

S3 is not designed for querying objects metadata in bulk. You need to follow answer from @Frédéric Henri if you have less number of files in your bucket, else you need to store metadata in some other datastore like DynamoDB, whenever you put a file in the S3 bucket you can trigger Lambda to copy metadata to DynamoDB table and then you can query DynamoDB for metadata in whatever way you desire.

Upvotes: -1

Frederic Henri
Frederic Henri

Reputation: 53703

you need aws s3api head-object which does exactly what you want

The HEAD operation retrieves metadata from an object without returning the object itself. This operation is useful if you're only interested in an object's metadata. To use HEAD, you must have READ access to the object.

aws s3api head-object --bucket <mybucket> --key <value>

It is also available in nodeJS API, see http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#headObject-property

The method is called once per key, if you need to call for multiple items you need to cupple that with other method, lets say as an example you need all metadata for all your csv files from your bucket you would run

aws s3 ls --recursive <mybucket> \
| grep ".csv$" | cut -c 32- \
| xargs -I {} aws s3api head-object --bucket <mybucket> --key {}

The first part of the command list all files from your bucket, you grep only for a particular extension, strip out characters so you get only the key, and pass this key as argument of the head-object command

Upvotes: 2

Related Questions