CODEWITHSUNDEEP
php
Ziom Ziomalski
Ziom Ziomalski

Reputation: 115

PHP memory limit not reached but OOM killer kills php script

I'm executing long running php script (PHP 7.3) which is parsing XML's in a loop and adds data to MySQL. I've set memory limit to 512MB with ini_set('memory_limit', '512M') inside the script. Problem is that after parsing about half of xml's OOM killer kills php with error:

[Mon May 6 16:52:09 2019] Out of memory: Kill process 12103 (php) score 704 or sacrifice child
[Mon May 6 16:52:09 2019] Killed process 12103 (php) total-vm:12540924kB, anon-rss:12268740kB, file-rss:3408kB, shmem-rss:0kB

I've tried debugging the code with memory_get_usage and php_meminfo. They both show that script do not exceed 100MB of memory at start and end of each loop (xml's have the same size). I'm already unsetting all possible vars at end of each loop.

It looks like PHP used 12.5GB of RAM inspite of the 0.5GB memory limit in the script. I'm expecting PHP to throw a fatal memory exhausted error if memory limit is reached but it never happens.

Any ideas how can I debug this problem?

Upvotes: 2

Views: 2400

Answers (2)

Andrei Nikolaenko
Andrei Nikolaenko

Reputation: 1074

I have recently met this problem and the trick was to allow PHP as little memory as possible, say 8MB. Otherwise the system triggered an out-of-memory error before PHP did. It did not provide any info so I did not know which part of the script was causing it.

But with 8MB memory limit I got a PHP exception with line number and additional info. It was trying to allocate a rather big chunk of memory (about 200k) whereas nothing in my script was demanding it.

Following the line number it became immediately obvious that one of the functions went recursive thus causing infinite memory consumption.

Upvotes: 1

Eli
Eli

Reputation: 984

I have worked importing large xml files, and the require more memory that most people expect. is not only the memory use for the file, but related variables and processes, as @NigelRen suggested, the best way to handle large xml is reading the file in parts. Here is a simple example that I hope can give you an idea on how to do this. 

    $reader = new \XMLReader(); 
    //https://www.php.net/manual/en/book.xmlreader.php

    // path to the file, the LIBXML_NOCDATA will help if you have CDATA in your 
    // content
    $reader->open($xmlPath, 'ISO-8859-1', LIBXML_NOCDATA); 

    while ($reader->read()) {
        if ($reader->nodeType == XMLReader::ELEMENT) {

                try {

                    $xmlNode = new \SimpleXMLElement($reader->readOuterXml());
                    // do what ever you want

                } catch (\Throwable $th) {
                  // hanlde error
               }
        }
    }

Upvotes: 0

Related Questions