CODEWITHSUNDEEP
phpsymfonymemorymemory-managementdoctrine-orm
DrKey
DrKey

Reputation: 3495

Doctrine flush changes and free up memory

I'm experiencing problems working on a Symfony project with low memory limit. I need to create several entities, add them to another entity (entities are associated) and then flush changes to database. This happens in a while loop.

The problem is that sometimes the loop can't finish the whole cycle because the process crashes for memory exhausting. Of course I can't raise the memory limit.

I try to explain you better with some code...

Entities structure

class Device
{
    /**
     * @ORM\ManyToMany(targetEntity="Resource", inversedBy="devices",cascade={"persist", "refresh"}, fetch="EXTRA_LAZY")
     */
    protected $resources;

    public function addResource(Resource $resource)
    {
        if ($this->resources->contains($resource)) return;
        $this->resources->add($resource);
        $resource->addDevice($this);
    }
}

class Resource
{
    /**
     * @ORM\ManyToMany(targetEntity="Device", mappedBy="resources")
     */
    protected $devices;

    public function addDevice(Device $device)
    {
        if ($this->devices->contains($device)) return;
        $this->devices->add($device);
        $device->addResource($this);
    }
}

The loop

function doIt(Device $device) {
    while(...) {
        $res = new Resource(...);
        $device->addResource($res); // store $res in memory until end
    }
    $em->persist($device);
    $em->flush();
}

So the problem is that until the loop finishes, $device keeps in memory Resource objects and sometimes memory limit is reached.

What I tried so far

function doIt(Device $device) {
   $i = 0;
   while(...) {
       $res = new Resource(...);
       $device->addResource($res);
       $i++;
       if ($i % 100 == 0) {
              $em->persist($device);
              $em->flush($device);
              $device->setResources(null); // free up Resources in memory
              $em->refresh($device); // Resources are not stored in memory
       }
   }
   $em->persist($device);
   $em->flush();
}

This works not that good even because I see memory raising anyway (probably caused then by Doctrine). Moreover there are a lot of entities and several associations between them, and checking results in database gives me something wrong (e.g. an Entity not persisted).

What would you do?

Upvotes: 3

Views: 6982

Answers (2)

mlivan
mlivan

Reputation: 661

I never persist more then 50 entity after flush. Try to decrease $i to 50 or less. Maybe that help.

Upvotes: 1

Joe
Joe

Reputation: 2436

There are a few things you can try to improve the situation:

Disable doctrine SQL logging mechanism in the setup stage of your import

$this->getContainer()->get('doctrine')->getConnection()->getConfiguration()->setSQLLogger(null);

Try calling a clear periodically (i do it after a batch of items is processed and flushed), but read the docs before because there may be side effects depending on your code

$this->entityManager->clear();

You already seem to be clearing all unused objects and variables you could also try calling a manual gc_collect_cycle in php

gc_collect_cycles();

If your import is executed with a symfony Command make sure you start it with the --no-debug flag

Worked fine with an import command within my project although it didn't achieve complete memory neutrality but the remaining leaks were negligible. Maybe this can help in your case.

Upvotes: 8

Related Questions