CODEWITHSUNDEEP
c#performancefilebinary
Majak
Majak

Reputation: 1633

Slow reading hundreds of files

I have problem with processing of more binary files. I have many many folders, in each there is about 200 bin files.

I choose 2 of these directories, then all bin files (their paths) from these 2 directories save to List, and make some filtering with this list. At the end of this, in list is about 200 bin files.

Then I'm iterating over all filtered files, and from each read first 4x8 Bytes (I tried FileStream or BinaryReader). All this operations take about 2-6 seconds, but only for the first time. Next time it's fast enough. If nothing happens with files for a long time (about 30 minutes), the problem appears again.

So probably it's something about caching or what?

Can someone help me please? Thanks

Upvotes: 2

Views: 374

Answers (2)

RB.
RB.

Reputation: 37222

One possibility is that your drive is going to sleep (typically a drive will be configured to power down after 15-30 minutes). This can add a significant delay (5 seconds would be a typical figure) as the hard-drive is span back up to speed.

Luckily, this is an easy thing to test. Just set the power-down time to, say, 6 hours, and then test if the behaviour has changed.

Upvotes: 0

dutzu
dutzu

Reputation: 3920

It is very possible that the handles to the files are disposed and that's why after a while the GC removes them and it takes longer or simply that the files are loaded in RAM by the OS and then it serves them to you from there and that's why it is faster, but that is not the issue, the process runs slow because it is slow, it isn't relevant that it is faster the 2nd time because you mustn't rely on that.

What i suggest is to parallellise as much as possible the processing of those files to be able to harness the full power of the hardware at hand.

Start by isolating the code that handles a file and then run the code within a Parallel.ForEach and see if that helps.

Upvotes: 0

Related Questions