CODEWITHSUNDEEP
c#.netfile-iomemory-mapped-files
Ehsan
Ehsan

Reputation: 1682

What is the fastest way to write hundreds of files to disk using C#?

My program should write hundreds of files to disk, received by external resources (network) each file is a simple document that I'm currently store it with the name of GUID in a specific folder but creating hundred files, writing, closing is a lengthy process.

Is there any better way to store these amount of files to disk? I've come to a solution, but I don't know if it is the best.

First, I create 2 files, one of them is like allocation table and the second one is a huge file storing all the content of my documents. But reading from this file would be a nightmare; maybe a memory-mapped file technique could help. Could working with 30GB or more create a problem?

Edit: What is the fastest way to storing 1000 text files on disk ? (write operation performs frequently)

Upvotes: 1

Views: 2263

Answers (3)

ChrisLively
ChrisLively

Reputation: 88064

The first thing you should do is profile your app. In particular you want to get the counters around Disk Queue Length. Your queue length shouldn't be any more than 1.5 to 2 times the number of disk spindles you have.

For example, if you have a single disk system, then the queue length shouldn't go above 2. If you have a RAID array with 3 disks, it should be more than 6.

Verify that you are indeed write bound. If so then the best way to speed up performance of doing massive writes is to buy disks with very fast write performance. Note that most RAID setups will result in decreased performance.

If write performance is critical, then spreading out the storage across multiple drives could work. Of course, you would have to take this into consideration for any app that that needs to read that information. And you'll still have to buy fast drives.

Note that not all drives are created equal and some are better suited for high performance than others.

Upvotes: 2

Garo Yeriazarian
Garo Yeriazarian

Reputation: 2533

This is similar to how Subversion stores its repositories on disk. Each revision in the repository is stored as a file, and the repository uses a folder for each 1000 revisions. This seems to perform rather well, except there is a good chance for the files to either become fragmented or be located further apart from each other. Subversion allows you to pack each 1000 revision folder into a single file (but this works nicely since the revisions are not modified once created.

If you plan on modifying these documents often, you could consider using an embedded database to manage the solid file for you (Firebird is a good one that doesn't have any size limitations). This way you don't have to manage the growth and organization of the files yourself (which can get complicated when you start modifying files inside the solid file). This will also help with the issues of concurrent access (reading / writing) if you use a separate service / process to manage the database and communicate with it. The new version of Firebird (2.5) supports multiple process access to a database even when using an embedded server. This way you can have multiple accesses to your file storage without having to run a database server.

Upvotes: 2

Uwe Keim
Uwe Keim

Reputation: 40736

What about using the ThreadPool for that?

I.e. for each received "file", enqueue a write function in a thread pool thread that actually persists the data to a file on disk.

Upvotes: 0

Related Questions