CODEWITHSUNDEEP
httpamazon-s3limitetag
Walden Leverich
Walden Leverich

Reputation: 4546

How long can an ETag be (storing AWS S3 ETag in database)

Assuming I want to store an ETag in a database column, what length should I allocate?

As far as I can tell there is not limit on the length of an ETag in the spec (https://www.rfc-editor.org/rfc/rfc7232#section-2.3). Even if I use a varchar(max) technically someone could use more than 2billion characters in an ETag, but we know that's not realistic. We also know web servers will barf on more than a few KB in total headers (Maximum on HTTP header values?) so the limit is way lower than that.

Typically ETags are going to be hashes (don't have to be, '5' is a perfectly valid etag), so I'm thinking 64 bytes is a minimum (SHA512), 100 is probably 'safe'. But does anyone have a better limit? What have people seen in the wild?

(I actually only care about AWS S3 ETag values if someone has a answer for that specific case I'll take it)

Upvotes: 2

Views: 784

Answers (1)

dove
dove

Reputation: 20694

Was looking at doing the same myself and AWS do not specify, indeed, it "might or might not ne an MD5 digest of the object data". https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html#ETag

I'm assuming that the size of ETag is going to stay linearly the same, or roughly, depending on the size of the objects. The size will depend on part size configuration in S3 but I have kept with defaults and in fact am uploading to the S3 clone by CloudFlare that is called R2. Note that a hash is required for each part. So with a very large file and small parts you might get a very long ETag.

Long story short is that I too am going to make a good guess but going to leave a wider column to allow for changes and indeed bigger images, which is what I'm uploading.

The size I'm going for is 256 as seems a natural choice. It is on the large size but I'm working with a table that is purely for storing this so don't have much space constraints.

Upvotes: 0

Related Questions