CODEWITHSUNDEEP
symfonydoctrine-ormdoctrinebatch-processing
David Rojo
David Rojo

Reputation: 2404

How to fully reset doctrine for performance

I have a Symfony application with Doctrine.

I need to create a command that will iterate through several thousands of records from the database, retrieving related data and processing it.

As entities are loaded, doctrine becomes more and more slow - as its identity map increases - from dozens of records per second processed to barely a few records per minute.

I would expect that executing $em->clear() would reset doctrine and would allow my process to be more efficient but it generates the following exception when processing more results after the clear().

[Doctrine\ORM\ORMInvalidArgumentException]
A new entity was found through the relationship 'App\Entity\XXXX' that was not configured to cascade persist operations for entity: XXXXXX.

To solve this issue: Either explicitly call EntityManager#persist() on this unknown entity or configure cascade persist this association in the mapping for example @ManyToOne(..,cascade={"persist"})

My code is something like this:

$qb = $repository->createQueryBuilder....;
$i = 0;
do {
    $list = $this->paginator->paginate($qb->getQuery(), $i, 10);
    foreach($list as $item){
        $this->processItem($item);
    }
    $this->em->clear();
    $i++;
} while(count($list) > 0);

If I try to detach just $item in each iteration, it works, but the related entities loaded during processItem are not detached, so IdentityMap keeps growing.

It is like clear() doesn't really clear all, or that after clear() not all entities are properly reloaded.

Of course, if I remove the clear() function the error is not fired.

Upvotes: 0

Views: 1372

Answers (1)

David Rojo
David Rojo

Reputation: 2404

After digging and checking twice the code, thanks to @NicolaiFröhlich, i've found the problem.

The issue indeed was that upper in my code I was loading some entities, that then after clear() were removed from the Identity Map, and those entities where used in the processItem function.

Upvotes: 1

Related Questions