CODEWITHSUNDEEP
mysqlfile-ioserver-load
Bobbi X
Bobbi X

Reputation: 197

Can massive writing & deleting files hurt our server performance?

We run a system that for cache purposes, currently writes and deletes about 1,000 small files (10k) every hour. In the near future this number will raise to about 10,000 - 20,000 files being written and deleted every hour.

For every files that is being written a new row on our mysql DB is added and deleted when the file is deleted an hour later.

My question:

  1. Can this excessive write & delete operation hurt our server performance eventually somehow? (btw we currently run this on a VPS and soon on a dedicated server.)
  2. Can writing and deleting so many rows eventually slow our DB?

Upvotes: 1

Views: 224

Answers (2)

Ghigo
Ghigo

Reputation: 2332

You should make some calculations and estimate needed throughtput for the storage. In your worst scenario, 20000 files x 10K = 200MB per hour which is a very low requirement. Deleting a file, on modern filesystems, takes a very little time.

In my opinion you don't have to worry, especially if your applications creates and deletes files sequentially.

Consider also that modern operative systems cache parts of file system in memory to improve performance and reduce disk access (this is true especially for multiple deletes).

Your database will grow but engines are optimized for it, no need to care about it.

Only downside is that handling many small files could cause disk fragmentation if your file system is subjected to it.

For a performance bonus, you should consider to use a separate phisical storage for these files (e.g. a different disk drive or disk array) so you will take advantage of full bandwidth transfer with no other interferences.

Upvotes: 1

Mikhail Vladimirov
Mikhail Vladimirov

Reputation: 13890

This greatly depends on operating system, file system and configuration of file system caching. Also this depends on whether your database is stored on the same disk as files that are written/deleted.

Usually, operation that affect file system structure such as file creations and file deletions require some synchronous disk IO, so operating system will not loose these changes after power failure. Though, some operating systems and file systems may support more relaxed policy for this. For example, UFS file system on FreeBSD has nice "soft updates" option that does this. Probably etx3/Linus should have similar feature.

Once you will move to dedicated server I think it would be reasonable to attach several HDDs to it and to make sure that database is stored on once disk while massive file operations are performed on another disk. In this cases DB performance should not be affected.

Upvotes: 1

Related Questions