CODEWITHSUNDEEP
amazon-web-servicesoptimizationamazon-s3storage
Simon Jan
Simon Jan

Reputation: 22

Optimize photo storage nomenclature on Amazon S3

I have to store lots of photos (+1 000 000, one max 5MB) and I have a database, every record has 5 photos, so what is the best solution:

I use Amazon S3 server.

Upvotes: 0

Views: 116

Answers (2)

Mausam Sharma
Mausam Sharma

Reputation: 892

i would suggest you to name your photos like this while uploading in batch:

user1/image1.jpeg

user2/image2.jpeg

Though these names would not effect the way objects are stored on s3 , these names will simply be 'keys' of 'objects', as there is no folder like hierarchical structure in s3 , but doing these will make objects appear in folders which will help to segregate images easily if you want later to do so.

for example , let us suppose you stored all images with unique names and you are using unique UUID to map records in database to images in your bucket.

But later on suppose you want all 5 photos of a particular user, then what will you have to do is

  1. scan the database for particular username
  2. Retrieve UUID's for the images of that user
  3. and then using the UUID for fetching images from s3

But if you name images by prefixing username to it , you can directly fetch images from s3 without making any reference to your database.

For example, to list all photos of user1, you can use this small code snippet in python :

import boto3
s3 = boto3.resource('s3')
Bucket=s3.Bucket('bucket_name')
for obj in Bucket.objects.filter(Prefix='user1/'):
    print(obj.key)

while if you don't use any user-id in key of object , then you have to refer database to do a mapping between photos and records even just to get a list of images of a particular user

Upvotes: 1

John Rotenstein
John Rotenstein

Reputation: 269470

A lot of this depends on your use-case, such as how the database and the photos will be used. There is not enough information here to give a definitive answer.

However, some recommendations for the storage side...

The easiest option is just to use a UUID for each photo. This is effectively a random name that has no meaning. Store that name in your database and your system will know which image relates to which record. There is no need to ever rename the images because the names are just Unique IDs and convey no further information.

When you want to provide access to a particular image, your application can generate an Amazon S3 pre-signed URL that grants time-limited access to an object. After the expiry time, the URL does not work so the object remains private. Granting access in this manner means that there is no need to group images into directories by "owner", since access is granted per-object rather than per-owner.

Also, please note that Amazon S3 doesn't actually support folders. Rather, the Key ("filename") of the object is the entire path (eg user-2/foo.jpg). This makes it more human-readable (because the objects 'appear' to be in folders), but doesn't actually impact the way data is stored behind-the-scenes.

Bottom line: It doesn't really matter how you store the images. What matters is that you store the image name in your database so you know which image matches which record. Avoid situations where you need to rename images - just give them a name and keep it.

Upvotes: 1

Related Questions