CODEWITHSUNDEEP
amazon-web-servicesamazon-s3
init_js
init_js

Reputation: 4611

S3 Eventual consistency: Is it safe to delete an object while there's an outstanding copy for it?

I'm issuing an S3 CopyObject (aka PUT copy) from a source bucket that is different than the destination bucket. I'm wondering if it's safe to delete the source after CopyObject has returned OK to the REST client. By "safe", I mean that the destination object will eventually show up, and that it will initially contain all of the data available at the time the copy was issued.

Corruption in the copy is perhaps unlikely (given that most operations are atomic), but the two operations cancelling each other might be possible. I wish the docs were slightly more fleshed out wrt to eventual consistency.

(in my scenario, nothing's writing to the source nor destination key in the interim. and the same client does the copy and delete).

e.g. synchronous pseudocode:

try:

    # make sure this is a create. read-after-create consistency
    my_tmpname = "new_tempfile" + uuid4()
    s3_put(data, "s3://my-bucket1/" + my_tmpname)

    ...

    # copy it to its final location
    s3_copy("s3://my-bucket1/new_tempfile", "s3://my-bucket2/final_location")
finally:
    # Cleanup temp file.
    #
    # Can this delete interfere with the copy in flight?
    # e.g. Should one wait a few seconds/minutes?
    # e.g. Should one ensure that the target exists before deleting source?
    s3_delete("s3://my-bucket/new_tempfile")

Upvotes: 1

Views: 1802

Answers (2)

Michael - sqlbot
Michael - sqlbot

Reputation: 179124

Yes, it is perfectly safe to delete an object immediately after a successful copy operation using that object as the source object, because copy operations are not asynchronous.

The copy request does not return until the operation has either succeeded or failed.

To help better ensure data durability, Amazon S3 PUT and PUT Object copy operations synchronously store your data across multiple facilities before returning SUCCESS.

https://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html

The consistency model in S3 is related only to the visibility of objects, not the durability of their storage.

Upvotes: 3

Julio Faerman
Julio Faerman

Reputation: 13501

Amazon S3 features read-after-write consistency for PUTs and the copy case you exemplified:

Amazon S3 Data Consistency Model

Amazon S3 provides read-after-write consistency for PUTS of new objects in your S3 bucket in all regions with one caveat. The caveat is that if you make a HEAD or GET request to the key name (to find if the object exists) before creating the object, Amazon S3 provides eventual consistency for read-after-write.

Check the documentation for consistency of the other operations.

https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html

Upvotes: 0

Related Questions