CODEWITHSUNDEEP
amazon-web-servicesamazon-s3
MIDE11
MIDE11

Reputation: 3530

Amazon S3 / Merge between two buckets

I have two buckets a and b. Bucket b contains 80% of the objects in a.

I want to copy the remain 20% objects which in a into b, without downloading the objects to local storage.

I saw the AWS Command Line Interface, but as I understant, it copy all the objects from a to b, but as I said - I want that it will copy only the files which exist in a but doesn't exists in b.

Upvotes: 3

Views: 6612

Answers (2)

Mohammad Ghonaim
Mohammad Ghonaim

Reputation: 434

Install aws cli and configure it with access credentials

Make sure both buckets have the same directory structure

AWS S3 docs

The following sync command syncs objects under a specified prefix and bucket to objects under another specified prefix and bucket by copying s3 objects. A s3 object will require copying if the sizes of the two s3 objects differ, the last modified time of the source is newer than the last modified time of the destination, or the s3 object does not exist under the specified bucket and prefix destination. In this example, the user syncs the bucket mybucket2 to the bucket mybucket. The bucket mybucket contains the objects test.txt and test2.txt. The bucket mybucket2 contains no objects:

aws s3 sync s3://mybucket s3://mybucket2

Upvotes: 6

Daniel Payne
Daniel Payne

Reputation: 125

You can use the AWS SDK and write a php or other supported language script that will make a list of filenames from both buckets, use array_diff to find out the files that are not common, then copy the files from Bucket A to memory then place the file in Bucket B.

This is a good place to start: https://aws.amazon.com/sdk-for-php/

More in depth on creating arrays of filenames (keys): [http://docs.aws.amazon.com/AmazonS3/latest/dev/ListingObjectKeysUsingPHP.html][2]

Some code for retrieving keys

$objects = $s3->getIterator('ListObjects', array('Bucket' => $bucket));

foreach ($objects as $object) {
    echo $object['Key'] . "\n";
}

Here describes how to move keys from bucket to bucket

// Instantiate the client.
$s3 = S3Client::factory();

// Copy an object.
$s3->copyObject(array(
    'Bucket'     => $targetBucket,
    'Key'        => $targetKeyname,
    'CopySource' => "{$sourceBucket}/{$sourceKeyname}",
));

You are going to want to pull the keys from both buckets, and do an array_diff to get a resulting set of keys that you can then loop through and transfer. Hope this helps.

Upvotes: 0

Related Questions