CODEWITHSUNDEEP
phpmemory-managementmemory-leaksout-of-memory
Chloe
Chloe

Reputation: 26294

How do I optimize memory usage in PHP for this function?

This function is passed about 70k objects to process. It has no problem loading the array, and it gets through about half the iterations before it fails. Memory is limited to ini_set('memory_limit','465M'); (cloud service). It always fails in the $googleFunction: /app/vendor/googleads/googleads-php-lib/src/Google/Api/Ads/Common/Lib/AdsSoapClient.php.

I've already tried passing the array as a reference, switching from array_chunk to using an array_slice at a time, passing the $chunk by reference to the $googleFunction, unsetting the $chunk at the end, and calling gc_collect_cycles() after each iteration.

How can it still fail? There must be a memory leak somewhere, but there are no large assignments aside from $chunk and $result, and each function called goes out of scope during each iteration, so anything it may have allocated is supposed to be garbage collected. I feel it may have something to do with the function references. After about a quarter of the iterations, it uses 240M. It grows by about 10M per iteration. 

  function googleJob($job = null, &$list, $googleFunction, $before = null) {
    // initialize $total, $gaw, $processed
    for ($chunkIndex = 0; $chunkIndex < count($list); $chunkIndex += 2000) { 
      echo '3:'.memory_get_usage().' ';
      $chunk = array_slice($list, $chunkIndex, 2000); # limit of 2000/request
      echo '4:'.memory_get_usage().' ';
      if ($before) {
        foreach ($chunk as $item) {
          $before($item); # function reference 
        }
      }
      echo '5:'.memory_get_usage().' ';

      $i = 0; // try harder to make Google work
      $result = null;
      do {
        try {
          $result = $gaw->$googleFunction($chunk);
          echo '6:'.memory_get_usage().' ';
        } catch (\SoapFault $e) { # try to remove the bad items and try again
          // no errors generated
        }
      } while ($result == null && $chunk); // Retry the request if not empty

      array_walk($chunk, function($item) { $item->save(); });
      echo '7:'.memory_get_usage().' ';
      $processed += count($chunk);
      if ($job) {
        $job->progress = round($processed / $total * 100);
        $job->save() or Yii::error($job->getErrors());
      }
      echo '8:'.memory_get_usage().' ';      
      unset($chunk);
      unset($result);
      echo '9:'.memory_get_usage().'... ';
      gc_collect_cycles();
      echo memory_get_usage().PHP_EOL;
    }
  }

Memory 'profiling' output:

3:110267832 4:110372112 5:111920328 6:123908368 7:129432080 8:129432080 9:121662520... 121662520
3:121662520 4:121766800 5:123281704 6:138001000 7:143493888 8:143493888 9:135745264... 135745264

Upvotes: 2

Views: 2313

Answers (1)

volkinc
volkinc

Reputation: 2128

It seems to me you are abusing soap service. if you telling us that your code fails at $googleFunction I can offer you set the $chunk with 100 or 200 objects.

The second thing is a $item->save() function. If you have an access to the class you should check if there are HAS-IS classes. The only place PHP leak the memory is like this construction:

class Foo {
    function __construct()
    {
        $this->bar = new Bar($this);
    }
}

class Bar {
    function __construct($foo = null)
    {
        $this->foo = $foo;
    }
}



for($i = 0; $i < 10; $i++){

     $foo = new Foo();
     unset($foo);
     echo number_format(memory_get_usage()) . "<br>";

}

so If you have objects, I suspect Active Record objects, created in a save() function don't forget to unset them. the easy way is to add destruct like this:

function __destruct()
    {
        unset($this->bar);
    }

This should help

Upvotes: 1

Related Questions