CODEWITHSUNDEEP
perlmemorymemory-leaks
tomypro
tomypro

Reputation: 95

Questions around memory utilization in Perl

SO community,

I have been scratching my head lately around two memory issues I am running into with some of my perl scripts and I am hoping I am finding some help/pointers here to better understand what is going on.

Questionable observation #1:

I am running the same perl script on different server instances (local laptop macosx, dedicated server hardware, virtual server hardware) and am getting significantly varying results in the traced memory consumption. Just after script initialization one instance would report be a memory consumption of the script of 210 MB compared to 330 MB on another box which is a fluctuation of over 60%. I understand that the malloc() function in charge of "garbage collection" for Perl is OS specific but are there deviations normal or should I be looking more closely at what is going on?

Questionable observation #2:

One script that is having memory leaks is relatively trivial:

foreach(@dataSamples) {

  #memorycheck_1

  my $string = subRoutine($_);
  print FILE $string;

  #memorycheck_2

}

All variables in the subRoutine are kept local and should be out of scope once the subroutine finishes. Yet when checking memory usage at #memorycheck_1 and #memorycheck_1 there is a significant memory leak.

Is there any explanation for that? Using Devel::Leak it seems there are leaked pointers which I have a hard time understanding where they would be coming from. Is there an easy way to translate the response of Devel::Leak into something that can actually give me pointers from where those leaked references origin?

Thanks

Upvotes: 3

Views: 323

Answers (2)

DavidO
DavidO

Reputation: 13942

The most common reason for a memory leak in Perl is circular references. The simplest form would be something along the lines of:

sub subRoutine {
    my( $this, $that );
    $this = \$that;
    $that = \$this;
    return $_[0];
}

Now of course people reading that are probably saying, "Why would anyone do that?" And one generally wouldn't. But more complex data structures can contain circular references pretty easily, and we don't even blink an eye at them. Consider double-linked lists where each node refers to the node to its left and its right. It's important to not let the last explicit reference to such a list pass out of scope without first breaking the circular references contained in each of its nodes, or you'll get a structure that is inaccessible but can't be garbage collected because the reference count to each node never falls to zero.

Per Eric Strom's excellent suggestion, the core module Scalar::Util has a function called weaken. A reference that has been weakened won't hold a reference count to the entity it refers to. This can be helpful for preventing circular references. Another strategy is to implement your circular-reference-wielding datastructure within a class where an object method explicitly breaks the circular reference. Either way, such data structures do require careful handling.

Another source of trouble is poorly written XS modules (nothing against XS authors; it's just really tricky to write XS modules well). What goes on behind the closed doors of an XS module may be a memory leak.

Until we see what's happening inside of subRoutine we can only guess whether or not there's actually an issue, and what the source of the issue may be.

Upvotes: 3

Sinan Ünür
Sinan Ünür

Reputation: 118156

You have two different questions:

1) Why is the memory footprint not the same across various environments?

Well, are all the OS involved 64 bit? Or is there a mix? If one OS is 32 bit and the other 64 bit, the variation is to be expected. Or, as @hobbs notes in the comments, is one of the perls compiled with threads support whereas another is not?

2) Why does the memory footprint change between check #1 and check #2?

That does not necessarily mean there is a memory leak. Perl won't give back memory to the OS. The memory footprint of your program will be the largest footprint it reaches and will not go down.

Neither of these points is Perl specific. For more detail, you'll need to show more detail.

See also Question 7.25 in the C FAQ and further reading mentioned in that FAQ entry.

Upvotes: 3

Related Questions