CODEWITHSUNDEEP
perlmemory-management
InnovWelt
InnovWelt

Reputation: 660

Need advice for Memory usage in perl

I am not new to perl , but I couldn't able to solve this usecase.

Here is my problem statement:

I am using Activestate perl 5.12 32-bit. I want to hold some 32 x 8MB array of double in Memory at a time. unfortunately, I am getting 'out of memory' error in this case. for example, the code below will lead to out of memory.

my $aref1 = [(.25) x (8*1024*1024)];
my $aref2 = [(.25) x (8*1024*1024)];
my $aref3 = [(.25) x (8*1024*1024)];
my $aref4 = [(.25) x (8*1024*1024)];
my $aref5 = [(.25) x (8*1024*1024)];
my $aref6 = [(.25) x (8*1024*1024)];
my $aref7 = [(.25) x (8*1024*1024)];
my $aref8 = [(.25) x (8*1024*1024)];
my $aref9 = [(.25) x (8*1024*1024)];
my $aref10 = [(.25) x (8*1024*1024)];

Is there any way to efficiently handle it?

Note: In any case, the access to the arrays is needed at any time of execution as fast as possible (since the user cannot wait soo long)

the options I have tried:

  1. DBM::Deep - It takes more time
  2. PDL - Gives 'out of memory' error for 32 x 8MB array of double

Awaiting your valuable suggestions!

Upvotes: 4

Views: 274

Answers (3)

Sinan Ünür
Sinan Ünür

Reputation: 118156

Here is what I get on my Windows XP SP3 system using ActiveState Perl 5.14.2. Task manager was showing Commit charge: 778M/3958M when I ran the following script:

#!/usr/bin/env perl

use strict;
use warnings;

use Devel::Size qw(total_size);

my $unit = 1024 * 1024;
my $topj = (8 * $unit) - 1;
my @data;

for my $i (0 .. 31) {
    print "$i: ";
    my @row;
    $#row = $topj;
    for my $j (0 .. $topj) {
        $row[$j] = 0.25;
    }
    push @data, \@row;
    printf "%.0f\n", total_size(\@data)/$unit;
}

Output:

C:\temp> yy
0: 224
1: 448
2: 672
3: 896
4: 1120
5: 1344
6: 1568
7: 1792
Out of memory!
8:

On the other hand, the following C program does better:

#include <stdlib.h>
#include <stdio.h>

#define ROWSIZE 8*1024*1024

int main(void) {
    int n = 1;
    while (calloc(ROWSIZE, sizeof(double))) {
        printf("%d: success!\n", n);
        n += 1;
    }
    return 0;
}

Output:

1: success!
2: success!
3: success!
…
26: success!
27: success!

for the same 1.7GB footprint.

Upvotes: 2

ikegami
ikegami

Reputation: 386551

32 arrays * 8 Mi-doubles/array * 8 bytes/double = 2 GiB.

32-bit processes on Windows only have 2GiB of usable address space. Your raw data would take up all available address space, leaving nothing for the data structure's overhead, perl, your program and other variables. The only way it's going to fit in memory is if you switch to a 64-bit Perl. Otherwise, you will have to contend with a necessarily slower solution.

Upvotes: 11

Leon Timmermans
Leon Timmermans

Reputation: 30235

This will create a perl value for each element, which will be quite heavy. You might want to look at something that stores the values as doubles, such as Tie::CArray or Tie::Array::PackedC.

Upvotes: 3

Related Questions