CODEWITHSUNDEEP
phpmysqlphotostorage
Kenny
Kenny

Reputation: 685

Folder Structure for storing millions of images?

I am building a site that is looking at Millions of photos being uploaded easily (with 3 thumbnails each for each image uploaded) and I need to find the best method for storing all these images.

I've searched and found examples of images stored as hashes.... for example...

If I upload, coolparty.jpg, my script would convert it to an Md5 hash resulting in..

dcehwd8y4fcf42wduasdha.jpg

and that's stored in /dc/eh/wd/dcehwd8y4fcf42wduasdha.jpg but for the 3 thumbnails I don't know how to store them

QUESTIONS..

  1. Is this the correct way to store these images?

  2. How would I store thumbnails?

  3. In PHP what is example code for storing these images using the method above?

Upvotes: 15

Views: 9397

Answers (5)

Shail
Shail

Reputation: 1575

Improve Answer.

For millions of Images, as yes, it is correct that using database will slow down the process

The best option will be either use "Server File System" to store images and use .htaccess to add security.

or you can use web-services. many servers like provide Images Api for uploading, displaying. You can go on that option also. For example Amazon

Upvotes: -3

Mihai Iorga
Mihai Iorga

Reputation: 39704

How am I using the folder structure:

  • I'm uploading the photo, and move it like you said:

    $image = md5_file($_FILES['image']['tmp_name']);
// you can add a random number to the file name just to make sure your images will be "unique"
$image = md5(mt_rand().$image);
$folder = $image[0]."/".$image[1]."/".$image[2]."/";

// IMAGES_PATH is a constant stored in my global config
define('IMAGES_PATH', '/path/to/my/images/');
// coolparty = f3d40fc20a86e4bf8ab717a6166a02d4
$folder = IMAGES_PATH.$folder.'f3d40fc20a86e4bf8ab717a6166a02d4.jpg';
// thumbnail, I just append the t_ before image name
$folder = IMAGES_PATH.$folder.'t_f3d40fc20a86e4bf8ab717a6166a02d4.jpg';
// move_uploaded_file(), with thumbnail after process
// also make sure you create the folders in mkdir() before you move them

  • I do believe is the base way, of course you can change the folder structure to a more deep one, like you said, with 2 characters if you will have millions of images.

Upvotes: 11

sberry
sberry

Reputation: 132018

The reason you would use a method like that is simply to reduce the total number of files per directory (inodes).

Using the method you have described (3 levels deeps) you are very unlikely to reach even hundreds of images per directory since you will have a max number of directories of almost 17MM. 16**6.

As far as your questions.

  1. Yeah, that is a fine way to store them.

  2. The way I would do it would be

    /aa/bb/cc/aabbccdddddddddddddd_thumb.jpg
    /aa/bb/cc/aabbccdddddddddddddd_large.jpg
    /aa/bb/cc/aabbccdddddddddddddd_full.jpg

    or similar

  3. There are plenty of examples on the net as far as how to actually store images. Do you have a more specific question?

Upvotes: 7

Robbie
Robbie

Reputation: 17710

If you're talking millions of photos, I would suggest you farm these off to a third party such as Amazon Web Services, more specifically for this Amazon S3. There is no limit for the number of files and, assuming you don't need to actually list the files, there is no need to separate them into directories at all (and if you do need to list, you can use different delimeters and prefixes - http://docs.amazonwebservices.com/AmazonS3/latest/dev/ListingKeysHierarchy.html). And your hosting/rereival costs will probably be lower than doing yourself - and they get backed up.

To answer more specifically, yes, split by sub directories; using your structure, you can drop the first 5 characters of the filename as you alsready have it in the directory name.

And thumbs, as suggested by aquinas, just appent _thumb1 etc to the filename. Or store in separate folders themsevles.

Upvotes: 2

aquinas
aquinas

Reputation: 23786

1) That's something only you can answer. Generally, I prefer to store the images in the database so you can have ONE consistent backup, but YMMV.

2) How? How about /dc/eh/wd/dcehwd8y4fcf42wduasdha_thumb1.jpg, /dc/eh/wd/dcehwd8y4fcf42wduasdha_thumb2.jpg and /dc/eh/wd/dcehwd8y4fcf42wduasdha_thumb3.jpg

3) ??? Are you asking how to write a file to the file system or...?

Upvotes: -3

Related Questions