CODEWITHSUNDEEP
amazon-web-servicesamazon-s3command-line
user1083828
user1083828

Reputation: 43

AWS s3 sync to upload if file does not exist in target

I have uploaded about 1,000,000 files from my local directory to s3 buckets/subfolders and some of them have failed.

I would like to use the 'sync' option to capture those that did not make it the first time. The s3 modified date is the date/time my file was uploaded (which differs from my source file date/times).

As I understand, sync will upload a file to the target if it does not exist, if the file date has changed, or if the size is different.

Can I modify the command line to NOT use the file date as a consideration for syncing? I ONLY want to copy a file if it does not exist.

aws s3 sync \localserver\localshare\folder s3://mybucket/Folder1

Upvotes: 1

Views: 3419

Answers (1)

Tsigalko2003
Tsigalko2003

Reputation: 86

aws s3 sync will compare the "last modified time".

For the objects in S3, there is only one timestamp LastModified, which should be when you uploaded the files.

For your local file (assume a posix linux file system). It should have 3 timestamps: last-access, last-modified, last-status-change. Only last-modified time will be used for comparison.

Now support you uploaded 1M files and some of them failed. For all the files had uploaded successfully, they should have identical last-modified time, and then another sync will not upload them again (sync will validate whether those files are identical and it will be considerable long for the validations for 1M objects.)

On the meantime, you can use aws s3 sync --size-only arguments. It fits what you described. But be sure to check whether it is really something you need. I mean, in many cases, many files could be keep the same size even after being modified (intentionally or accidentally), --size-only will ignore such same-size files.

Upvotes: 6

Related Questions