CODEWITHSUNDEEP
c#.netfilestreamdiskhard-drive
J4N
J4N

Reputation: 20727

.Net Write continuously data to the disk in different files

We have an application that extract data from several hardware devices. Each device's data should be stored in a different file.

Currently we have one FileStream by file and doing a write when a data comes and that's it.

We have a lot of data coming in, the disk is struggling with an HDD(not a SSD), I guess because the flash is faster, but also because we do not have to jump to different file places all the time.

Some metrics for the default case: 400 different data source(each should have his own file) and we receive ~50KB/s for each data(so 20MB/s). Each data source acquisition is running concurrently and at total we are using ~6% of the CPU.

Is there a way to organize the flush to the disk in order to ensure the better flow?

We will also consider improving the hardware, but it's not really the subject here, since it's a good way to improve our read/write

Upvotes: 3

Views: 619

Answers (1)

usr
usr

Reputation: 171178

Windows and NTFS handle multiple concurrent sequential IO streams to the same disk terribly inefficiently. Probably, you are suffering from random IO. You need to schedule the IO yourself in bigger chunks.

You might also see extreme fragmentation. In such cases NTFS sometimes allocates every Nth sector to each of the N files. It is hard to believe how bad NTFS is in such scenarios.

Buffer data for each file until you have like 16MB. Then, flush it out. Do not write to multiple files at the same time. That way you have one disk seek for each 16MB segment which reduces seek overhead to near zero.

Upvotes: 3

Related Questions