CODEWITHSUNDEEP
performanceperl
Qiang Li
Qiang Li

Reputation: 10855

Why does the execution time of my perl code vary so widely?

I found the following perl code executed in surprisingly varying speeds, sometimes fast, sometimes very slow. I have a few folders containing tens of thousands of files, which I need to run this code through. I am running this on cygwin with windows 7. Just wonder if someone could please help me to speed it up, or as least to figure out why the speed is varying. My CPU and memory should be plentiful in all these situations.

outer loop to iterate through a list of $dir's

opendir(DIR, $dir);
@all=readdir(DIR);
@files = (0..$#all);
$i=-1;
foreach $current (@all){     
    if (-f "$dir/$current") {
        $files[++$i]=$current;
    }
}
push @Allfiles,@files[0..$i];
closedir(DIR);

Upvotes: 1

Views: 312

Answers (2)

Mat
Mat

Reputation: 206689

You're probably I/O bound, so changes to your code probably won't affect the total runtime - runtime will be affected by whether the directory entries are in cache or not.

But your code uses temporary arrays for no good reason, using too much RAM if the directories are very large. You could simplify it to:

opendir(DIR, $dir);
while (my file = readdir(DIR)) {
    push @Allfiles, $file if (-f "$dir/$file");
}
closedir(DIR);

No temporary arrays.

Upvotes: 7

ObscureRobot
ObscureRobot

Reputation: 7336

If it is slow the first time you run, and fast after that, then the problem is that your system is caching the reads. The first time you run your code, data has to be read off your disk. After that, the data is still cached in RAM. If you wait long enough, the cache will flush and you will have to hit the disks again.

Or sometimes you may be running some other disk intensive task at the same time, but not at other times when you run your code.

Upvotes: 4

Related Questions