CODEWITHSUNDEEP
perl
lazy
lazy

Reputation: 77

Limitations of the Perl Tie::File module

I tried using the Tie:File module to write a text file which should be containing 1 billion lines, but it throws an error after writing 16 million

"Out of memory!"

"Callback called exit at C:/perl/lib/Tie/File.pm line 979 during global destruction."

this is the code I tried with.

    use Tie::File;

    tie @array, 'Tie::File', "Out.txt";

    $start = time();

    for ($i = 0; $i <= 15000000; $i++) {
       $array[$i].= "$i,";
    }

    $end = time();

    print("time taken: ", $end - $start, "seconds");
    untie @array;

I don't know why it throws an error. Any solutions to overcome this? It also took about 55 minutes to write 16 million records and it throws error! Is this usual time it takes to write?

Upvotes: 2

Views: 890

Answers (1)

Borodin
Borodin

Reputation: 126742

The Tie:File module is known to be quite slow, and it is best used where the advantage of having random access to the lines of the file outweighs the poor performance.

But this isn't a problem with the module, it is a limitation of Perl. Or, more accurately, a limitation of your computer system. If you take the module out of the situation and just try to create an ordinary array with 1,000,000,000 elements then Perl will die with an Out of memory! error. The limit for my 32-bit build of Perl 5 version 20 is around 30 million. For 64-bit builds it will be substantially more.

Tie:File doesn't keep the whole file in memory but pages it in and out to save space, so it can handle very large files. Just not that large!

In this case you don't have any need of the advantages of Tie:File, and you should just write the data sequentially to the file. Something like this

use strict;
use warnings;
use 5.010;
use autodie;

open my $fh, '>', 'Out.txt';

my $time = time;

for my $i (0 .. 15_000_000) {
   print $fh "$i,\n";
}

$time = time - $time;

printf "Time taken: %d seconds\n", $time;

This program ran in seven seconds on my system.

Please also note use strict and use warnings at the start of the program. This is essential for any Perl program, and will quickly reveal many simple problems that you would otherwise overlook. With use strict in place, each variable must be declared with my as close as possible to the first point of use.

Upvotes: 4

Related Questions