CODEWITHSUNDEEP
mongodbmongoosecompose-db
PirateApp
PirateApp

Reputation: 6204

Mongodb bulk Insert connection timeout

I have a bulk operation where I upsert 10000 items every 2 hours in my mongodb database. The code for that looks like this

let bulk = models.Product.collection.initializeUnorderedBulkOp();
...
    if (bulk.length > 0) {
        bulk.find({
            "$or": [
                {
                    "updatedAt": {
                        "$lt": timestamp
                    }
                },
                {
                    "discount": {
                        "$eq": 0
                    }
                }
            ]

        }).remove()
        bulk.execute((error, result) => {
            if (error) {
                console.error('Error while inserting products' + JSON.stringify(error))
            }
            else {
                console.log('Successfully inserted ' + result.nInserted + ' upserted ' + result.nUpserted + ' matched ' + result.nMatched + ' modified ' + result.nModified + ' removed ' + result.nRemoved)
            }
        })
    }
    else {
        console.log('There were no bulk operations to execute ' + products.length)
    }
}

My connection keeps getting timed out. My options for the mongoose connection look like this

let options = {
    mongos: {
        ssl: true,
        sslValidate: true,
        sslCA: ca,
    }
}

I am well aware of this connection setting that is being discussed on other stackoverflow threads

server: {
    socketOptions: {
        keepAlive: 300000,
        connectTimeoutMS: 30000
    }
}

I read the documentation for keepAlive and connectTimeoutMS but how do I know the right value for both, Do I need the socketTimeoutMS as well?

Thank you for your advice in advance

UPDATE 1

I keep getting this error:

 {"name":"MongoError","message":"connection 0 to aws-ap-southeast-1-portal.2.dblayer.com:15284 timed out"}

My connection options look like this now //Options for compose.io database

let options = {
    mongos: {
        ssl: true,
        sslValidate: true,
        sslCA: ca,
    },
    server: {
        socketOptions: {
            keepAlive: 300000,
            connectTimeoutMS: 300000
        }
    },
    replset: {
        socketOptions:
        {
            keepAlive: 300000,
            connectTimeoutMS: 300000
        }
    }
}

Upvotes: 4

Views: 6508

Answers (2)

Zahid Riaz
Zahid Riaz

Reputation: 2889

Lowering batch was not much of help to me but instead increasing the connection timeout values.

Here is my connection string which solved the problem for me.

MONGO_URI=mongodb://user:[email protected]:27017/dbname?keepAlive=true&poolSize=30&autoReconnect=true&socketTimeoutMS=360000&connectTimeoutMS=360000

Upvotes: 4

molokoloco
molokoloco

Reputation: 4602

Ok, i have lowered my bulk operation by batch of 100 instead of 1000 and it work fine now.. It worked well for 1000 when i was doing :

bulk.insert(programme);

But now, i doing this, and it's not working for 1000/bulk :

bulk.find({ eventId: programme.eventId }).upsert().replaceOne(programme);

Upvotes: 1

Related Questions