CODEWITHSUNDEEP
phpimage
Arnold
Arnold

Reputation: 495

single folder or many folders for storing 8 million images of hundreds of stores?

I am developing a price comparison site with hundreds of stores and thousands of product images for each store.

In total there are 8 million images.

Each image has this file name format: StoreID-productID.jpg

Each image size is less than 10KB. average 7KB.

The site is developed with php and mysql.

I have linux dedicated server.

I have a lot of disk space. so, that is not an issue.

My questions are:

  1. Should I keep all the images of all the stores in a single folder?

So, 8 million images will be in a single folder.

Will it effect efficiency in retrieving them?

At present I am using this method and would like to know if there are any disadvantages of this method.

  1. Should I keep a folder for each store and keep the images of that store in that folder?

like: 

images/storeID1/
images/storeID2/
....

Please suggest. Thank you

Upvotes: 3

Views: 2370

Answers (2)

akatakritos
akatakritos

Reputation: 9858

Depends on the file system you are using. (EXT3, NTFS, FAT, etc).

Each will have different folder size limits and performance characteristics.

For 8 million files t will be safest to separate as many folders as you can. Then you have the option to make them different physical drives if you run into scaling issues.

If you're on Linux, its easy to mount another drive right among your existing folder structure. You could alternatively use a symbolic link.

images/Store1/FILES...
images/Store2 ---> /mount/SDA01/Store2/ (symbolic link to a separate drive)

Limits

See this SuperUser question for more detail about different File System limits: https://superuser.com/questions/446282/max-files-per-directory-on-ntfs-vol-vs-fat32

Note these are the absolute limits of what the system can handle. Performance at the upper bound of those limits will definitely suffer.

Upvotes: 4

EmmanuelG
EmmanuelG

Reputation: 1051

On one of my recent projects I had to store a large number of images as they came in over time. We decided that the best thing to do would be to seperate them into numbered directories with a set number of images in each directory. Our database would store the file name, directory, foreign key of the item it was associated with, a flag to allow us to disable it without needig to go and delete stuff, and the date it was downloaded. It's worked great since.

Upvotes: 3

Related Questions