CODEWITHSUNDEEP
phpmongodb
Mohamed Atef
Mohamed Atef

Reputation: 167

Iam trying to insert 1 milion row in mongodb (PHP)

I'm trying to insert 1 million records in mongodb it take so much time while many says that it can do it so fast tell me please what's wrong with my code i'm using php to do that 

$m = new MongoClient();
$db = $m->mydb;
echo "Database mydb selected";

$collection = $db->createCollection("mycol");
echo "Collection created succsessfully";

$collection = $db->mycol;
echo "Collection selected succsessfully";
for ($i=0; $i<1000000; $i++)
{
    $document = array( 
        "title" => "MongoDB".$i, 
        "description" => "database", 
        "likes" => 100,
        "url" => "http://www.tutorialspoint.com/mongodb/",
        "by" => "tutorials point"
     );

     $collection->insert($document);
}

echo "Document inserted successfully";

Upvotes: 1

Views: 1754

Answers (2)

chridam
chridam

Reputation: 103375

You can leverage your insert operations by using the Bulk Operations API which is available from MongoDB 2.6 and greater. The current PHP driver should support these methods, where in particular you need the MongoWriteBatch class for insert operations, in particular create an Insert Write Batch with the MongoInsertBatch class:

<?php
    $mc = new MongoClient("localhost");
    $collection = $mc->selectCollection("test", "category");

    $batch = new MongoInsertBatch($collection);
    $counter = 0;

    for($i=0; $i<1000000; $i++) {

        $document = array( 
            "title" => "MongoDB".$i, 
            "description" => "database", 
            "likes" => 100,
            "url" => "http://www.tutorialspoint.com/mongodb/",
            "by", "tutorials point"
        );
        $batch->add($document);
        $counter++;

        if ( $counter % 1000 === 0 ) {
            $ret = $batch->execute(array("w" => 1));
            $counter++;
            $batch = new MongoInsertBatch($collection);        
        }
    }

    if ( $counter > 0 ) {
        $ret = $batch->execute(array("w" => 1));
    }
?>

In the above, each document created in the for loop is added to an insert operation which is then added to a batch via the .add() method, and is only sent to the server on the call to .execute(). Ordered write write operations are sent to the server, in the order provided, for serial execution. If a write fails, any remaining operations will be aborted.

There is some management of the "size" of these operations implemented with the modulo operation, which should be generally handled by the driver but keeps this in manageable sizes should you want to check the write result.

The key here is that rather than waiting for the write response from the server for every single update, the operations are sent and responded to in "batches". The reduction in "overhead" here when performing mass updates is considerable as there is less "back and forth" when communicating with the server.

Upvotes: 2

Will
Will

Reputation: 24699

Use Batched / Bulk Inserts instead:

<?php
$m = new MongoClient();

$db = $m->mydb;
echo "Database mydb selected";
$collection = $db->createCollection("mycol");

$db = $m->mydb;
echo "Database mydb selected";
$collection = $db->mycol;

$batchSize = 1000;
$documents = array();
for ($i=0; $i<1000000; $i++)
{
      $documents[] = array( 
            "title" => "MongoDB".$i, 
            "description" => "database", 
            "likes" => 100,
            "url" => "http://www.tutorialspoint.com/mongodb/",
            "by" => "tutorials point"
      );

      // If we've reached `$batchSize` entries, perform the batched insert.
      if (($i+1) >= $batchSize)
      {
            $collection->batchInsert($documents);
            $documents = array();
      }
}

// Finish the last batch.
if (!empty($documents))
{
      $collection->batchInsert($documents);
}

Here, we insert 1000 records at a time (adjustable via $batchSize). This requires MongoDB 2.6 or greater.

Upvotes: 3

Related Questions