CODEWITHSUNDEEP
amazon-web-servicesamazon-s3synchronization
user33991
user33991

Reputation: 61

How does aws s3 sync handle interruptions? Is it possible that files are corrupted?

I want to perform an aws s3 sync to a bucket. What happens with the files if the sync gets aborted manually? Is it possible hat there is a corrupt file left behind? AWS says that multipart-upload is used for files >5G and here corrupt files cannot occur. But what about files smaller than 5GB?

I couldnt find exact information in aws documentation about that. I want to use aws s3 sync and not aws s3api.

Upvotes: 0

Views: 913

Answers (1)

varnit
varnit

Reputation: 1897

AWS S3 is not a hierarchical filesystem. It is devided into two significant components, the backing store and the index which, unlike in a typical filesystem, are separate... so when you're writing an object, you're not really writing it "in place." Uploading an object saves the object to the backing store, and then adds it to the bucket's index, which is used by GET and other requests to fetch the stored data and metadata for retrieval. Hence in your case if the sync is aborted then its AWS responsibility to delete that file and it would not be indexed,

Coming to the multipart uploads, here also aws would not list the complete file until you send the last part of your multipart upload, you can also send an abort request to abort the multipart upload in that case aws would stop charging you for your partially uploaded files.

for more information about multipart upload refer to this document:

S3 multipart upload

Upvotes: 1

Related Questions