CODEWITHSUNDEEP
performancefilefilesystemsoperating-system
Rawhi
Rawhi

Reputation: 6403

The max number of files in one directory?

Is there any limitation on number of files in one directory (in any host) ?
If I have a directory with 30k (named from 1 to 3ok) files and another one with only 10 is there a major difference in performance to fetch a specific files ?
thanx

Upvotes: 1

Views: 1863

Answers (2)

Dmitry Zagorulkin
Dmitry Zagorulkin

Reputation: 8548

it depends from your file system type. Answer on this question will be find out in your current file system type spec.

Archlinux Wiki perfomance optimizing page

Summary:

  • XFS: Excellent performance with large files. Low speed with small files. A good choice for /home. -Reiserfs: Excellent performance with small files. A good choice for /var.
  • Ext3: Average performance, reliable.
  • Ext4: Great overall performance, reliable,has performance issues with sqlite and some other databases.
  • JFS: Good overall performance, very low CPU usage, extremely fast resume after power failure.
  • Btrfs: Probably best overall performance (with compression) and lots of features. Still in heavy development and fully supported, but considered as unstable. Do not use this filesystem yet unless you know what you are doing and are prepared for potential data loss.

fsck time vs Inode Count

Upvotes: 3

kratenko
kratenko

Reputation: 7592

I'd say the max number of files is OS specific and file system specific. But having a huge number ob files in one directory can drastically hit your performance, when accessing a file.

I can not give you any numbers for any specific os/fs, but maybe a solution if you have performance issues:

In mediawiki software (thats the software wikipedia runs on) they use subdirectories to counter that problem. This is how they store media files:

  1. md5-hash the name of the file
  2. take the first digit of the md5hash as subdirectory of the files dir
  3. take the first 2 digits of the md5hash as name of a subsubdirectory to that subdirectory
  4. store the file there

this way they can find the file by the name only, but don't need to rely on a good os/fs for sillions of files. It results in something like this:

http://upload.wikimedia.org/wikipedia/commons/7/74/Flag_of_Hamburg.svg is the path for Flag_of_Hamburg.svg

Upvotes: 2

Related Questions