CODEWITHSUNDEEP
mongodb
adnaan.zohran
adnaan.zohran

Reputation: 809

Concurrent writes in mongodb

When I am inserting/updating a document in a collection, is the lock applied on the database or the collection. Suppose I have two collections and they are independant of each other in the same database and wants to do write operations on them concurrently. Is this possible?

Here is the code I am using to test this: 

var assert      = require('assert'),
    MongoClient = require('mongodb').MongoClient,
    async       = require('async');

var station_list    = require('./station_list.json'),
    trains_list     = require('./trains_list.json');

var stationList = [],
    trainsList  = [];

var MONGO_URL   = 'mongodb://localhost:27017/test';

for(var i=0; i<station_list.stations.length; i++)
    stationList.push(station_list.stations[i].station_code);

for(var i=0; i<trains_list.trains.length; i++)
    trainsList.push(trains_list.trains[i].code);

console.log('trains : ' + trainsList.length + ' stations : ' + stationList.length);

populateTrains();
populateStations();

function populateTrains() {
    async.eachSeries(trainsList, populateTrainDb, function (err) {
        assert.equal(null, err);        
    });
}

function populateTrainDb(code, callback) {
    MongoClient.connect(MONGO_URL, function (err, db) {
        assert.equal(null, err);
        var jsonData = {};
        jsonData.code = code;
        db.collection('trainsCon').replaceOne(
            {'code' : code}, jsonData, {upsert: true, w:1}, function (err, res) {
                assert.equal(null, err);
                db.close();
                callback();
            });
    });
}

function populateStations() {
    async.eachSeries(stationList, populateStationDb, function (err) {
        assert.equal(null, err);
    });
}

function populateStationDb(code, callback) {
    MongoClient.connect(MONGO_URL, function (err, db) {
        assert.equal(null, err);
        var jsonData = {};
        jsonData.code = code;
        db.collection('stationsCon').replaceOne(
            {'code' : code}, jsonData, {upsert:true, w:1}, function (err, res) {
                assert.equal(null, err);
                db.close();
                callback();
            });
    });
}

The two json files : station_list.json and trains_list.json have around 5000 entries. So after running the given program I get this error after a while : 

C:\Users\Adnaan\Desktop\hopSmart\node_modules\mongodb\lib\server.js:242
        process.nextTick(function() { throw err; })
                                      ^
 AssertionError: null == { [MongoError: connect EADDRINUSE 127.0.0.1:27017]
  name: 'MongoError',
  message: 'connect EADDRINUSE 127.0.0.1:27017' }
    at C:\Users\Adnaan\Desktop\hopSmart\testing.js:52:10
    at C:\Users\Adnaan\Desktop\hopSmart\node_modules\mongodb\lib\mongo_client.js:276:20
    at C:\Users\Adnaan\Desktop\hopSmart\node_modules\mongodb\lib\db.js:224:14
    at null.<anonymous> (C:\Users\Adnaan\Desktop\hopSmart\node_modules\mongodb\lib\server.js:240:9)
    at g (events.js:273:16)
    at emitTwo (events.js:100:13)
    at emit (events.js:185:7)
    at null.<anonymous> (C:\Users\Adnaan\Desktop\hopSmart\node_modules\mongodb-core\lib\topologies\server.js:301:68)
    at emitTwo (events.js:100:13)
    at emit (events.js:185:7)

When I check the number of entries entered the database, around 4000 entries had already been entered in both the collections. So what I get from the above experiment was that an error might have occured when one write was being attempted while inside other collection a document must have been getting written.

So how should I proceed to have this concurrency without conflicting locks.

Upvotes: 1

Views: 11223

Answers (3)

Andrey
Andrey

Reputation: 121

You should not be calling MongoClient.connect every time. That's causing a ton of connections to open and close all the time which is overloading mongo. You should let the MongoClient manage the connection pool. Change it so that you store the db object from MongoClient.connect. Something like this:

var db

MongoClient.connect(url, function(err, database){
    db = database;
}

Upvotes: 0

Saleem
Saleem

Reputation: 8978

I don't think it's concurrency issue with MongoDB, but I could be driver or even with test itself. I have created a sample application couple of weeks ago to stress test MongoDB while working on a nasty bug. I used C# and MongoDB 3.0 on Windows 10. I have inserted million of documents in multithreaded environment but couldn't crash MongoDB. 

Parallel.For(0, 10000, (x =>
            {
                var lstDocs = new List<BsonDocument>();
                for (var i = 0; i < 100; i++)
                {
                    lstDocs.Add(new BsonDocument(doc));
                }

                collection.InsertMany(lstDocs);
                lstDocs.Clear();

            }));

You can find code in gist here.

Upvotes: 0

Nicolas
Nicolas

Reputation: 2231

The answer to this question can be quite long and depends on various factors (MongoDB version, storage engine, type of operations you are doing, sharding, etc.). I can only recommend you to read carefully the Concurrency section of the MongoDB documentation, and in particular the lock granularity part.

Make sure to choose the right version of MongoDB first as the behaviour varies greatly from one version to another (e.g. database locking in pre-3.0 vs. collection locking for most operations in post-3.0 using NMAPv1).

Upvotes: 6

Related Questions