CODEWITHSUNDEEP
linuxperformancefilesystems
Drakosha
Drakosha

Reputation: 12165

Estimation or measurement of amount of iops to create a file

I'd like to know how many I/O operations (iops) does it take to create an empty file. I am interested in linux and GFS file system, however other file systems information is also very welcome.

Suggestions how to accurately measure this would be also very welcome.

Real scenario (requested by answers):

  1. Linux
  2. GFS file system (if you can estimate for another - pls do)
  3. create a new file in existing directory, the file does not exist, using the following code
  4. Assume directory is in cache and directory depth is D

Code:

int fd = open("/my_dir/1/2/3/new_file", O_CREAT | S_IRWXU);
// assuming fd is valid
fsync(fd);

Upvotes: 1

Views: 1189

Answers (3)

Drakosha
Drakosha

Reputation: 12165

I did the following measurement, we wrote an application that creates N files as described in the question. We ran this application on a disk which was devoted to this application only, and measured IOps amount using iostat -x 1

The result, on GFS and linux kernel 2.6.18 is 2 IOps per file creation.

This answer is based on MarkR answer.

Upvotes: 0

MarkR
MarkR

Reputation: 63586

For an artifical measurement:

  1. Create a blank filesystem on its own block device (e.g. vmware scsi etc)
  2. Mount it, call sync(), then record the number of IOPS present on that block dev.
  3. Run your test program against the filesystem, and do no further operations (not even "ls").
  4. Wait until all unflushed blocks have been flushed - say about 1 minute or so
  5. Snapshot the iops count again

Of course this is highly unrealistic, because if you created two files rather than one, you'd probably find that there were less than twice as many.

Also creating empty or blank files is unrealistic - as they don't do anything useful.

Directory structures (how deep the directories are, how many entries) might contribute, but also how fragmented it is and other arbitrary factors.

Upvotes: 1

user82238
user82238

Reputation:

The nature of the answer to this question is; best case, normal case, worst case. There is no single answer, because the number of IOPS required will vary according to the current state of the file system. (A pristine file system is highly unrealistic scenario).

Taking FAT32 as an example, best case is 1. Normal case depends on the degree of file system fragmentation and the directory depth of the pathname for the new file. Worse case is unbounded (except by the size of the file system, which imposes a limit on the maximum possible number of IOPs to create a file).

Really, the question is not answerable, unless you define a particular file system scenario.

Upvotes: 0

Related Questions