CODEWITHSUNDEEP
phpsymfonydoctrine-ormmany-to-one
gawellus
gawellus

Reputation: 61

Doctrine clear() and many to one relation

I need to import large data from csv file (about 1m records). For avoid memory leaks I tried bulk insert from doctrine documentation:

$batchSize = 20;
for ($i = 1; $i <= 10000; ++$i) {

    //code

    $em->persist($user);
    if (($i % $batchSize) === 0) {
        $em->flush();
        $em->clear(); // Detaches all objects from Doctrine!
    }
}
$em->flush(); //Persist objects that did not make up an entire batch
$em->clear();

The problem is my insert contains "many to one" relations and everytime I use clear() object duplicate entries from this relation.

Is there any possibility to detach entity and avoid duplicates?

Upvotes: 1

Views: 1281

Answers (2)

Nikola Loncar
Nikola Loncar

Reputation: 2671

yes there is. You need to perform smart clean. Meaning that you provide entity type that you want to clear. Example :

$em->clear(SomeEntity::class);
$em->clear(OtherEntity::class);

this will remove from EM only those types and leave everything else. In that way you can reuse those other objects.

Upvotes: 1

Erwan
Erwan

Reputation: 41

Using an ORM for importing large data is pretty time and memory consuming. I never find a way of make a fast and sane import with Doctrine. Besides the import I made this way are a PITA to maintain.

After many years of importing data, the most effective way I found is :

  • CSV import in temporary table (LOAD DATA INFILE in MySQL or COPY in pgsql)
  • Make an insert or update query (INSERT INTO .. SELECT .. ON DUPLICATE KEY in MySQL)
  • Do the more specific thing in either SQL or PHP depending on the complexity

This way I can import dozen of big CSV files in seconds.

Hope it help.

Upvotes: 0

Related Questions