CODEWITHSUNDEEP
arraysperlpack
gangabass
gangabass

Reputation: 10666

A way to pack string into another light weight structure

i need a way to pack/zip a string (10 digit number) into another structure which will takes less memory.

My goal is to randomize a big array of numbers (112M of records) but i can't load it into memory right now (only 30M of records). So i think about representing each digit via 4 bits that way i can reduce the size of data structure two times. But i need more compression.

So i need some hints.

Thanks. Roman

Upvotes: 0

Views: 153

Answers (2)

ikegami
ikegami

Reputation: 385565

That's not that huge of a memory requirement. A 64-bit build of Perl should be able to handle it.

You could save memory if you avoid using an array of numbers. Create a string of them. That will cut down on the overhead.

$n = $a[$i];
$a[$i] = $n;

becomes

$n = unpack('L', substr($s, $i*4, 4));
substr($s, $i*4, 4) = pack('L', $n);

Yes, it will be slower. Here's are the savings:

use Devel::Size qw( total_size );
my @a = (1..1000);
say total_size \@a;       # 20036
my $s = qq{\0}x(4*1000);
say total_size \$s;       #  4028

And that's not counting all the overhead and fragmentation of the memory management system used by the array version.

NOTE: This only allows up to 4,294,967,296. Supporting higher numbers will require twice the memory.

Upvotes: 0

choroba
choroba

Reputation: 241758

If your numbers fit in long, you can use pack 'l' or pack 'L' (four bytes).

Upvotes: 1

Related Questions