CODEWITHSUNDEEP
javaamazon-s3
Detlef Brendle
Detlef Brendle

Reputation: 151

Amazon S3 file availability after upload

I want to upload a file during a testcase and then immediately download it to verify the file has been uploaded. I use a key which is already uplopaded to S3.

When my upload is completed I do the download but somehow I get the previous version of this file.

My code looks something like this:

final ObjectMetadata objectMetadata = new ObjectMetadata(); objectMetadata.setContentType(binaryDataReference.getContentType()); objectMetadata.setContentLength(data.length); objectMetadata.setContentEncoding(xmlEncoding.getCharset()); s3Client.putObject(BUCKET_NAME, key, new ByteArrayInputStream(data), objectMetadata); Thread.sleep(30000); // this I've done because I thought its a timing issue. S3Object object = s3Client.getObject(BUCKET_NAME, binaryDataKey);

But the Object I get is not the one I just uploaded.

Do I need to do some thing or does anybody have go an idea what is wrnong with this approach ?

Upvotes: 4

Views: 2399

Answers (1)

rhashimoto
rhashimoto

Reputation: 15869

It's just the way that S3 works:

Amazon S3 buckets in the US Standard region provide eventual consistency. Amazon S3 buckets in all other regions provide read-after-write consistency for PUTS of new objects and eventual consistency for overwrite PUTS and DELETES.

This means that after writing a new object you can always immediately read it as long as it is not in the US Standard region. Otherwise - i.e. if you are in the US Standard region or you are overwriting or deleting an existing object - you are only guaranteed that "eventually" the result of your action will be visible.

I once asked Amazon if after overwriting an object and succeeding in reading back the latest version, whether all subsequent reads on the same connection would be guaranteed to also return the latest version. That is, if I wrote version 2 and read version 2, would I never read version 1 again? They did not respond.

If you do require stronger consistency then Google Cloud Storage provides read-after-update and read-after-delete consistency, though GCS was slightly more expensive than S3 the last time I checked.

Upvotes: 3

Related Questions