CODEWITHSUNDEEP
mongodbdoctrine-odm
bazo
bazo

Reputation: 755

doctrine odm batch insert - not all documents get saved

i have a file of 67352 lines. one line is one name. i'm inserting these names in a batch using foreach. out of these 67352, only 52262 records get persisted to database when using batch size of 100. if the batch size is higher, fewer documents get persisted, and if it is lower, more documents get persisted, but i never get all of the documents persisted.

the version of odm is 1.0.0beta3, i tried this also with 1.0.0rc1, with the same result.

the code looks like this:

$batchSize = 100;
foreach ($data as $name)
{
    $doc = new \SomeDocument;
    $doc->setName($name);

    $dm->persist($doc);
    $i++;

    if ($i % $batchSize == 0)
    {
        $dm->flush();
        $dm->clear();
    }
}
$dm->flush();

can anyone tell me what's wrong? thanks

Upvotes: 0

Views: 1653

Answers (1)

Adam Comerford
Adam Comerford

Reputation: 21682

You haven't mentioned what you are using for an index etc. but let's assume it's the default and you are not running into uniqueness issues (especially since your results are variable). It sounds like some of your writes are failing/getting lost.

Try running the batch inserts with:

$dm->flush(array('safe'=>true));

It will be a bit slower but at least it should now throw an exception when you see a failure and you can take it from there.

Upvotes: 1

Related Questions