CODEWITHSUNDEEP
amazon-web-servicesamazon-s3aws-cli
Jonathan Andersson
Jonathan Andersson

Reputation: 1422

AWS S3 CLI CP file and add metadata

Trying to copy a local file named test.txt to my s3 bucket and add metadata to the file.

But it always prints error:

argument --metadata-directive: Invalid choice, valid choices are: COPY | REPLACE

Is it possible to do this with the cp command, as I understand the docs it should be possible. AWS CLI CP DOCS

This is the commands I've tried:

aws s3 cp test.txt to s3://a-bucket/test.txt --metadata x-amz-meta-cms-id:34533452

aws s3 cp test.txt to s3://a-bucket/test.txt --metadata-directive COPY --metadata x-amz-meta-cms-id:34533452

aws s3 cp test.txt to s3://a-bucket/test.txt --metadata-directive COPY --metadata '{"x-amz-meta-cms-id":"34533452"}'

aws s3 cp test.txt to s3://a-bucket/test.txt --metadata '{"x-amz-meta-cms-id":"34533452"}'

aws --version: aws-cli/1.9.7 Python/2.7.10 Darwin/16.1.0 botocore/1.3.7

OS: macOS Sierra version 10.12.1

Edit

Worth mentioning is that uploading a file without the --metadata flag works fine.

Hmm, I've checked the help for my version of cli with aws s3 cp help Turns out it does not list --metadata as an option, as the docs at the given link above does.

If runnig older version of aws cli

Use aws s3api put-object

How to upload a file to a bucket and add metadata:

aws s3api put-object --bucket a-bucket --key test.txt --body test.txt --metadata '{"x-amz-meta-cms-id":"34533452"}'

Docs: AWS S3API DOCS

Upvotes: 32

Views: 44851

Answers (4)

Frederic Henri
Frederic Henri

Reputation: 53713

Indeed the support for metadata option has been added since 1.9.10

aws s3 Added support for custom metadata in cp, mv, and sync.

so upgrading your aws cli to this version (or even better to latest) - and the metadata value needs to be a map so

aws s3 cp test.txt s3://a-bucket/test.txt --metadata '{"cms-id":"34533452"}'

Upvotes: 51

almeynman
almeynman

Reputation: 7398

There are special flags to set Content-Type and Content-Encoding

aws s3 cp test.gz. s3://a-bucket/test.gz --content-type application/octet-stream --content-encoding gzip

Upvotes: 3

Srini Sydney
Srini Sydney

Reputation: 580

Install s3cmd tools (free) and invoke like so:

s3cmd modify --add-header x-amz-meta-foo:bar s3://<bucket>/<object>

With x-amz-meta-foo:bar header you will get foo as key and bar as value of that key.

Upvotes: 7

kartik
kartik

Reputation: 2135

There is bug with metadata directive "COPY" option.

aws s3api copy-object --bucket testkartik --copy-source testkartik/costs.csv --key costs.csv --metadata-directive "COPY" --metadata "SomeKey=SomeValue"

enter image description here

Below are the three steps to understand cli command with JQ workaround.

  1. Install JQ library to deal with json metadata using command line.

  2. Read the existing metadata.

    aws s3api head-object --bucket <bucket> --key <key> | jq '.Metadata' | jq --compact-output '. +{"new":"metadata", "another" : "metadata"}'

  3. Add new metadata.

    aws s3api copy-object --bucket <bucket-name> --copy-source <bucket/key> --key <key> --metadata-directive "REPLACE" --metadata $(READ-THE-EXISTING-From-Step-2)

    Complete command in one go.

    aws s3api copy-object --bucket <bucket-name> --copy-source <bucket/key> --key <key> --metadata-directive "REPLACE" --metadata $(aws s3api head-object --bucket <bucket> --key <key> | jq '.Metadata' | jq --compact-output '. +{"new":"metadata", "another" : "metadata"}')

Upvotes: 0

Related Questions