CODEWITHSUNDEEP
laravelmemorycommandlaravel-5.6schedule
moses toh
moses toh

Reputation: 13172

How can I solve "Fatal error: Out of memory..." when I run command schedule laravel?

My command like this : 

<?php
namespace App\Console\Commands;
use Illuminate\Console\Command;
use App\Models\ItemDetail;
class ImportItemDetail extends Command
{
    protected $signature = 'sync:item-detail';
    protected $description = 'sync item detail';
    public function __construct()
    {
        parent::__construct();
        ini_set('max_execution_time', 0);
        ini_set('memory_limit', '-1');
    }
    public function handle()
    {
        $path = storage_path('json/ItemDetail.json');
        $json = json_decode(file_get_contents($path), true);
        foreach ($json['value'] as $value) {
            ItemDetail::updateOrCreate(
                [ 'entry_number' => $value['Entry_Number'] ],
                [ 
                    'item_number' => $value['Item_Number'],
                    'description' => $value['Description'],
                    ....
                ]
            );
        }
    }
}

If I run the commmand, there exist error like this :

Fatal error: Out of memory (allocated 255852544) (tried to allocate 8192 bytes)

Whereas I had set ini_set('memory_limit', '-1'); in construct

How can I solve the error?

Note :

My record in the json file contains hundreds of thousands of data. So it takes a long time to check the field

Update

My ItemDetail.json like this :

{
  "@odata.context":"http://myapp-svr01.www.myapp.com:1224/DynamicsNAV100/ODataV4/$metadata#Collection(NAV.ODATAITEM)","value":[
    {
      "@odata.etag":"W/\"JzE2O0lBQUFBQUNIbXdrQ0FBQAD5OzE4ODQ1MDQ4NjA7Jw==\"","Entry_Number":123,"Item_Number":"9805010101","Variant_Code":"039","Posting_Date":"2018-01-02T00:00:00Z","Entry_Type":"Sale","Description":"MISC BISBAN POLOS 11MM A847","Quantity":-7200,"Source_Type":"Customer","Order_Type":" ","Sales_Amount_Actual":1800000,"ETag":"16;IAAAAACHmwkCAAAA9;1884504860;"
    },{
      "@odata.etag":"W/\"JzE2O0lBQUFBQUNIbkFrQ0FBQAD5OzE4ODQ1MDQ5MTA7Jw==\"","Entry_Number":124,"Item_Number":"9805010102","Variant_Code":"100","Posting_Date":"2018-01-02T00:00:00Z","Entry_Type":"Sale","Description":"MISC BISBAN POLOS 11MM A915","Quantity":-7200,"Source_Type":"Customer","Order_Type":" ","Sales_Amount_Actual":1800000,"ETag":"16;IAAAAACHnAkCAAAA9;1884504910;"
    }
  ]
}

I only gave 2 records in json above. Actually there are around 150,000 records

Upvotes: 0

Views: 352

Answers (1)

Brian Lee
Brian Lee

Reputation: 18187

The problem is you're reading in the entire file at once, instead you should read it in chunks or line by line. For example:

$items = [];
$file = fopen(storage_path('json/ItemDetail.json'),'r');
while (!feof($file)){
    $line = fgets($file);
    $obj = json_decode($line);
    // $obj is type stdClass
    // transform $obj to fit the update or create method
    ItemDetail::updateOrCreate( ... )
}

update

You should look into a streaming parser when dealing with large files.

jsonstreamingparser would be a good choice.

Upvotes: 1

Related Questions