CODEWITHSUNDEEP
phpmemory-managementormmemory-leaksdoctrine
mkk
mkk

Reputation: 7693

doctrine 2.1: memory usage increased after flush

it is really unintuitive to me why after flush() the memory get increased instead of decreased? I cannot find anything useful in the documentation. I am flushing every 50 loop iterations and after each flush memory usage is being increased by 1MB. 100 flushing uses 100MB ! What am I missing here? I have seen on doctrine's website example with clear(), but it if I use it I got an exception that one of my object does not have cascade persist set. Could anyone explain me what is done behind the scenes and what should I do to properly unset all doctrine's objects?

Upvotes: 2

Views: 4829

Answers (3)

Sam S
Sam S

Reputation: 566

I had a similar situation with bulk inserting records (over 100K). I followed Rauni and Besnik suggestions and Doctrine's guide to batching but the memory still grew even after $em->clear() and resetEntityManager(). After a few days, I headed the information notice on Doctrine's documentation:

An ORM tool is not primarily well-suited for mass inserts, updates or deletions. Every RDBMS has its own, most effective way of dealing with such operations and if the options outlined below are not sufficient for your purposes we recommend you use the tools for your particular RDBMS for these bulk operations.

and used raw SQL (MySQL) and batched the inserts. Memory usage went from over 200MB with Doctrine down to 30MB with raw SQL. And I could see the memory usage decrease as I popped the elements to be inserted off of the source array. Also much quicker, from 220 seconds to 20 seconds.

Upvotes: 0

Rauni Lillemets
Rauni Lillemets

Reputation: 2407

Also, read here: http://docs.doctrine-project.org/en/2.0.x/reference/batch-processing.html

It suggests to use $em->clear() after every bulk of operations. (which should give the same effect as Besnik suggested).

From the previously mentioned link:

Bulk inserts in Doctrine are best performed in batches, taking advantage of the transactional write-behind behavior of an EntityManager. The following code shows an example for inserting 10000 objects with a batch size of 20. You may need to experiment with the batch size to find the size that works best for you. Larger batch sizes mean more prepared statement reuse internally but also mean more work during flush.

Upvotes: 5

Besnik
Besnik

Reputation: 6529

The problem is, that the entity manager handles all objects. You should reset it if you dont need the objects in ram after each X iterations. Try this out:

// get the entity manager and save objects and flush
$em = $this->get('doctrine')->getEntityManager();
...
$em->flush();

// then reset the entity manager
$this->get('doctrine')->resetEntityManager();

Upvotes: 4

Related Questions