CODEWITHSUNDEEP
mongodbpymongo
redice
redice

Reputation: 8747

Insert operation became very slow for MongoDB

The client is pymongo.

The program has been running for one week. It's indeed very fast to insert data before: about 10 million / 30 minutes.

But today i found the insert operation became very very slow.

There are about 120 million records in the goods collection now.

> db.goods.count()

123535156

And the indexs for goods collection is as following:

db.goods.getIndexes();

[
        {
                "v" : 1,
                "key" : {
                        "_id" : 1
                },
                "ns" : "shop.goods",
                "name" : "_id_"
        },
        {
                "v" : 1,
                "key" : {
                        "item_id" : 1,
                        "updated_at" : -1
                },
                "unique" : true,
                "ns" : "shop.goods",
                "name" : "item_id_1_updated_at_-1"
        },
        {
                "v" : 1,
                "key" : {
                        "updated_at" : 1
                },
                "ns" : "shop.goods",
                "name" : "updated_at_1"
        },
        {
                "v" : 1,
                "key" : {
                        "item_id" : 1
                },
                "ns" : "shop.goods",
                "name" : "item_id_1"
        }
]

And there is enough RAM and CPU.

Someone told me because there are too many records. But didn't tell me how to solve this problem. I was a bit disappointed with the MongoDB.

There will be more data needs to be stored in future(about 50 million new records per day). Is there any solution?

Met same situation on another sever(Less data this time, total about 40 million), the current insert speed is about 5 records per second.

> db.products.stats()
{
        "ns" : "c2c.products",
        "count" : 42389635,
        "size" : 554721283200,
        "avgObjSize" : 13086.248164203349,
        "storageSize" : 560415723712,
        "numExtents" : 283,
        "nindexes" : 3,
        "lastExtentSize" : 2146426864,
        "paddingFactor" : 1.0000000000132128,
        "systemFlags" : 1,
        "userFlags" : 0,
        "totalIndexSize" : 4257185968,
        "indexSizes" : {
                "_id_" : 1375325840,
                "product_id_1" : 1687460992,
                "created_at_1" : 1194399136
        },
        "ok" : 1
}

Upvotes: 3

Views: 7604

Answers (3)

czxttkl
czxttkl

Reputation: 496

I had a very similar problem.

First you need to make sure which is your bottleneck (CPU, memory and Disk IO). I use several unix tools (such as top, iotop, etc) to detect the bottleneck. In my case I found insertion speed was lagged by IO speed because mongod often took 99% io usage. (Note: my original db used mmapv1 storage engine).

My work around was to change storage engine to wiredtiger. (either by mongodump your original db then mongorestore into wiredtiger format, or start a new mongod with wiredtiger engine and then resync from other replica set memebers.) My insertion speed went to normal after doing that.

However, I am still not sure why mongod with mmapv1 suddenly drained IO usages after the size of documents reached a point.

Upvotes: 0

mohr_michael_a
mohr_michael_a

Reputation: 204

Another possible problem is IO. Depending on your scenario Mongo might be busy trying to grow or allocate storage files for the given namespace (i.e. DB) for the subsequent insert statements. If your test pattern has been add records / delete records / add records / delete records you are likely reusing existing allocated space. If your app is now running longer than before you might be in the situation I described.

Hope this sheds some light on your situation.

Upvotes: 0

rubenfa
rubenfa

Reputation: 851

I don't know if it is your problem, but take in mind that MongoDB has to update index for each insert. So if you have many indexes, and many documents, performance could be lower than expected.

Maybe, you can speed up inserts operations using sharding. You don't mention it in your question, so I guess you are not using it.

Anyway, could you provide us more information? You can use db.goods.stats(), db.ServerStatus or any of theese other methods to gather information about performance of your database.

Upvotes: 1

Related Questions