CODEWITHSUNDEEP
javagrailsamazon-web-servicesamazon-s3jclouds
Priyanshu Chauhan
Priyanshu Chauhan

Reputation: 5535

How do I update metadata for an existing Amazon S3 file?

I need to update the cache control header in all AmazonS3's Cloud Files. However, I can't figure out how I do that using the jclouds API. I'm using apache jclouds plugin. And I got two related answers:

The first answer is suggesting to use SwiftKey Api class which is not available in grails's jcloud plugin. The second answer is using AWS java sdk for which there is already a grails wrapping plugin https://grails.org/plugin/aws-sdk but it doesn't support metadata update.

Upvotes: 61

Views: 72987

Answers (8)

Chris Gilbert
Chris Gilbert

Reputation: 263

I couldn't find it documented anywhere, but if you are using Step Functions to make an SDK call to S3, you can set metadata like this, using MetadataDirective like with PHP and Ruby above (this example shows the parameters for s3:copyObject):

{
  "Bucket.$": "$.bucketName",
  "CopySource.$": "$.bucketSourceLocation",
  "Key.$": "$.destinationKey",
  "Acl": "bucket-owner-full-control",
  "ServerSideEncryption": "AES256",
  "MetadataDirective": "REPLACE",
  "Metadata": {
    "my-metadata.$": "$.my-metadata"
  }
}

Upvotes: 0

Damaon
Damaon

Reputation: 612

I had to do it recently and used JS for that:

const {
  S3Client,
  ListObjectsCommand,
  HeadObjectCommand,
  CopyObjectCommand,
} = require('@aws-sdk/client-s3');
const fs = require('fs');

const client = new S3Client({
  region: 'us-west-2',
});

const files = fs.readFileSync('images.txt', 'utf8').split('\n');

async function run() {
  for (const prefix of files) {
    await fixMetdata(prefix);
  }
}

const BUCKET_NAME = 'my-bucket';

async function fixMetdata(prefix) {
  // console.error(prefix);
  try {
    const listObjects = new ListObjectsCommand({
      Bucket: BUCKET_NAME,
      Prefix: prefix,
      MaxKeys: 1,
    });
    const data = await client.send(listObjects);

    if (!data.Contents) return;

    data.Contents.filter((f) => f.Key.includes('.jpg')).forEach(
      async (file) => {
        const fileMetadata = await client.send(
          new HeadObjectCommand({ Bucket: BUCKET_NAME, Key: file.Key })
        );
        // console.error(fileMetadata.ContentType);
        if (fileMetadata.ContentType !== 'image/jpeg') {
          const updatedMetadata = await client.send(
            new CopyObjectCommand({
              Bucket: BUCKET_NAME,
              Key: file.Key,
              CopySource: `${BUCKET_NAME}/${file.Key}`,
              MetadataDirective: 'REPLACE',
              ContentType: 'image/jpeg',
            })
          );
          console.log(file.Key);
        }
      }
    );
  } catch (err) {
    console.error('Error', err);
  }
}

run();

Upvotes: 3

Jellicle
Jellicle

Reputation: 30206

As stated in other answers, a "copy" request can be made which overwrites the existing record.

Ruby example

s3_client.copy_object({
  bucket: BUCKET, # Destination
  key: KEY, # Destination
  copy_source: "/#{BUCKET}/#{KEY}", # Source
  content_type: "text/html; charset=utf8", # Metadata update
  metadata_directive: "REPLACE"
})

Upvotes: 3

John Lindal
John Lindal

Reputation: 628

In my case, I care about the object versions, and copying creates a new version. I decided to create a parallel, "hidden" object with the mutable metadata. For example, if I have /foo/bar/baz, then I would create /foo/bar/.baz.

Upvotes: 0

styks
styks

Reputation: 3441

PHP Example

I understand this question was not PHP specific, but it may help someone as it is a top result on Google.

This will overwrite the existing object.

$client = new \Aws\S3\S3Client([
    'version' => '2006-03-01',
    'region' => 'BUCKET-REGION'
]);

$updateResponse = $client->copyObject([
    'Key' => 'OBJECTKEY',
    'Bucket' =>  'MYBUCKET',
    'CopySource' => 'MYBUCKET/OBJECTKEY',
    'MetadataDirective' => 'REPLACE',
    'Metadata' => [
        'width' => 441,
        'height' => 189
    ]
]);

The Content Type was an existing metadata item

Upvotes: 7

robnick
robnick

Reputation: 1768

As at March 2018, the latest version of the AWS SDK for .NET Core has changed. It now uses asynchronous programming. Many of the method signatures have changed. While you still cannot change metadata without the object copy solution that Dan has suggested, the code to do so has.

My solution is to update the existing S3 object with the modified metadata.

The following works for me to update a single metadata value (based on key and new value). I've got two loops for setting the metadata but it could be optimised to just have one:

string fileContents = string.Empty;
Dictionary<string, string> fileMetaData = null;

GetObjectRequest request = new GetObjectRequest
{
  BucketName = bucketName,
  Key = setKeyName
};

var response = await s3Client.GetObjectAsync(request);

// Read the contents of the file
using (var stream = response.ResponseStream)
{
  // Get file contents
  TextReader tr = new StreamReader(stream);
  fileContents = tr.ReadToEnd();
}

// Create the File Metadata collection
fileMetaData = new Dictionary<string, string>();
foreach (string metadataKey in response.Metadata.Keys)
{
  fileMetaData.Add(metadataKey, response.Metadata[metadataKey]);
}

// Update the metadata value (key to update, new value)
fileMetaData[metaDataKeyToUpdate] = metaDataNewValue;

// Update the existing S3 object    
PutObjectRequest putRequest1 = new PutObjectRequest
{
  BucketName = bucketName,
  Key = setKeyName,
  ContentBody = fileContents
};

// Set the metadata
foreach (string metadataKey in response.Metadata.Keys)
{
  putRequest1.Metadata.Add(metadataKey, fileMetaData[metadataKey]);
}

PutObjectResponse response1 = await s3Client.PutObjectAsync(putRequest1);

Upvotes: -3

Dan Gravell
Dan Gravell

Reputation: 8230

It is possible to change the metadata by performing an object copy (see How to update metadata using Amazon S3 SDK):

ObjectMetadata metadataCopy = new ObjectMetadata();
// copy previous metadata
metadataCopy.addUserMetadata("newmetadata", "newmetadatavalue");

CopyObjectRequest request = new CopyObjectRequest(bucketName, existingKey, bucketName, existingKey)
      .withNewObjectMetadata(metadataCopy);

amazonS3Client.copyObject(request);

Whether this is philosophically an "update" is up to you to decide.

Upvotes: 74

E.J. Brennan
E.J. Brennan

Reputation: 46841

You can't:

Each Amazon S3 object has data, a key, and metadata. Object key (or key name) uniquely identifies the object in a bucket. Object metadata is a set of name-value pairs. You can set object metadata at the time you upload it. After you upload the object, you cannot modify object metadata. The only way to modify object metadata is to make a copy of the object and set the metadata.

http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html

Upvotes: 39

Related Questions