CODEWITHSUNDEEP
amazon-web-servicesamazon-s3aws-cli
Yann
Yann

Reputation: 2522

How to get S3 object versions with metadata?

I use S3 versioning and would like to add some custom fields to the versions and retrieve it in the cheapest way.

Ideally, the custom fields should be represented in the output of an aws-cli command like this:

aws s3api list-object-versions --bucket mybucket --prefix myfile.md

without any additional n API calls. But I can't find if this endpoint can return metadata.

I can retrieve metadata only by individual version ID:

aws s3api head-object --bucket mybucket --version-id YgLFqd3oM_JBd2Dmwszxy.Up.fFX3xzi --key myfile.md

{
    "VersionId": "YgLFqd3oM_JBd2Dmwszxy.Up.fFX3xzi",
    "ContentLength": 2030,
    "AcceptRanges": "bytes",
    "ContentType": "application/x-yaml",
    "ETag": "\"4e29f3565aca5bf8d196bfabac536f1d\"",
    "LastModified": "Thu, 06 Jun 2019 15:54:22 GMT",
    "Metadata": {
        "comment": "some text"
    }
}

My current approach is to store metadata in DynamoDB with S3 object key and it's version as a primary key. But maybe it's an overhead and I can do everything with S3 only.

Upvotes: 2

Views: 3813

Answers (1)

Michael - sqlbot
Michael - sqlbot

Reputation: 178956

S3 does not have an API action that returns an object listing/object version listing as well as object metadata/object version metadata.

You can theoretically do everything with S3, you just have to iterate through the object listing and send a HEAD request for each object version -- it's easier with the SDKs than with the CLI -- but this isn't without costs in terns of per-request charges and execution time, so an external database may be preferrable. (I use one, but it uses RDS MariaDB instead of DynamoDB). S3 Event Notifications and AWS Lambda functions can maintain such an external index for you, so that you don't have to build it from the code that originally creates the objects.

Upvotes: 2

Related Questions