CODEWITHSUNDEEP
javamongodbspring-data
Bertie
Bertie

Reputation: 17697

How to speed up my mongodb test?

From my test, i get around 500 inserts, 200 queries, 400 updates per sec. I wonder on what i can tweak to increase these numbers.

I have read that others can achieve thousands or even tens of thousand of inserts in their tests, which is far better than my test. I wonder if i am missing something basic?

So, here are the facts :

The start time is : 2012-03-08 21:50:16

And i monitor using mongostat, and after reaching the time of 22:05:10, i terminate my unfinished application .. and here's the last output of the mongostat

insert  query update delete getmore command flushes mapped  vsize    res faults locked % idx miss %     qr|qw   ar|aw  netIn netOut  conn       time
   499    200    400      0       0     100       0  1023m  1.06g   581m    145      8.5          0       0|0     0|0   645k    97k     3   22:05:01
   503    201    403      0       0     102       0  1023m  1.06g   582m    154     10.7          0       0|0     0|1   651k    98k     3   22:05:02
   520    208    415      0       0     105       0  1023m  1.06g   582m    176     11.1          0       0|0     0|0   671k   101k     3   22:05:03
   504    202    403      0       0     102       0  1023m  1.06g   582m    167      7.2          0       0|0     0|0   651k    98k     3   22:05:04
   524    209    419      0       0     106       0  1023m  1.06g   582m    147      8.3          0       0|0     0|0   675k   102k     3   22:05:05
   534    213    428      0       0     107       0  1023m  1.06g   583m    176      7.4          0       0|0     0|0   690k   103k     3   22:05:06
   531    213    424      0       0     108       0  1023m  1.06g   584m    160      4.9          0       0|0     0|0   685k   104k     3   22:05:07
   533    213    427      0       0     107       0  1023m  1.06g   584m    164      6.9          0       0|0     0|0   689k   103k     3   22:05:08
   518    208    414      0       0     105       0  1023m  1.06g   585m    158      7.3          0       0|0     0|0   669k   101k     3   22:05:09
   521    208    417      0       0     105       0  1023m  1.06g   585m    154      4.7          0       0|0     0|0   673k   101k     3   22:05:10

And then i checked my insertion number :

> db.myCollection.find().size();
90575

And this is an example of my document inserted, which is also updated etc in the process

> db.myCollection.findOne().pretty();
{
        "_id" : "b146189a-56a4-4035-8245-c4bd6dc2bd22",
        "something1" : "my class is cool !",
        "something2" : {
                "value" : "this is a statement blah blah",
                "name" : "myStatement"
        },
        "something3" : {
                "size" : {
                        "value" : 0,
                        "name" : "size"
                },
                "value" : [
                        "6810cb0c-fa3e-4ca9-8a27-8432f2d1e828",
                        "a8276d05-a796-4c43-bc74-edc06d074099"
                ],
                "name" : "myids"
        },
        "something4" : {
                "myattr" : {
                        "value" : "something",
                        "name" : "name"
                },
                "attr" : {
                        "content" : {
                                "value" : "another another body body content content",
                                "name" : "content"
                        },
                        "contentId" : "b146189a-56a4-4035-8245-c4bd6dc2bd22",
                        "name" : "something"
                },
                "subsubchildchild" : {
                        "size" : {
                                "value" : 0,
                                "name" : "size"
                        },
                        "value" : [ ],
                        "name" : "subBodies"
                },
                "myId" : "b146189a-56a4-4035-8245-c4bd6dc2bd22",
                "name" : "hiccups"
        },
        "something5" : {
                "value" : false,
                "name" : "hahaha"
        },
        "something6" : {
                "name" : "okay this is just a test"
        },
        "something7" : {
                "value" : false,
                "name" : "remove me !"
        },
        "something8" : {
                "size" : {
                        "value" : 0,
                        "name" : "size"
                },
                "value" : [ ],
                "name" : "guess what"
        },
        "something9" : {
                "size" : {
                        "value" : 0,
                        "name" : "anotherSize"
                },
                "value" : [ ],
                "name" : "tarantula"
        },
        "something10" : {
                "value" : 8,
                "name" : "my exam score"
        },
        "something11" : {
                "size" : {
                        "value" : 0,
                        "name" : "justAnotherSize"
                },
                "value" : [ ],
                "name" : "myReference"
        },
        "something12" : {
                "size" : {
                        "value" : 0,
                        "name" : "size"
                },
                "value" : [ ],
                "name" : "myOtherReference"
        },
        "something13" : {
                "value" : "8b78fff0-50f5-4992-9972-89f9d944fee7",
                "name" : "user"
        },
        "something14" : {
                "dateTime" : "2012-03-08 21:50:17.480000000"
        },
        "something15" : {
                "value" : false,
                "name" : "lovely"
        }
}

And here's my db stat :

> db.stats();
{
        "db" : "qa",
        "collections" : 7,
        "objects" : 815197,
        "avgObjSize" : 622.2093211824872,
        "dataSize" : 507223172,
        "storageSize" : 610770944,
        "numExtents" : 57,
        "indexes" : 5,
        "indexSize" : 64197952,
        "fileSize" : 1056702464,
        "nsSizeMB" : 16,
        "ok" : 1
}

Also another question from curiosity .. judging from my main collection size, which has around 90k records, and other nonsubstantial collections which shouldnt be big in size, is it reasonable to have around 1TB of fileSize in this case ? Is there anything i can do to help reduce my filesize ?

Please share your thoughts.

Upvotes: 0

Views: 529

Answers (1)

Gates VP
Gates VP

Reputation: 45287

You seem to be getting a lot of faults on the mongostat. Any idea why?

Doing a combination of insertion and atomic updates like push, pull, inc, dec, set

How are you issuing these updates? By _id?

I have read that others can achieve thousands or even tens of thousand of inserts in their tests, which is far better than my test. I wonder if i am missing something basic?

According to mongostat you only have 3 connections active and your lock % is only about 10%.

  • Are you multi-threading your inputs?
  • Are you ruining this all on the same computer?
  • How is system IO?
  • Are you doing WriteConcern.Safe?

These are all considerations that could be affecting your throughput.

is it reasonable to have around 1TB of fileSize in this case ?

Base on your db.stats(), you only have about 600 MB of disk in use.

"storageSize" : 610770944 // = 610,770,944

Your average object size is 622 bytes, but you have 815,197 objects, not the 90k documents you claim to have.

Is there anything i can do to help reduce my filesize ?

Yes, reduce the size of the Keys in your JSON documents. For example:

"something1" : "my class is cool !"  => ~28 bytes
"s1": "my class is cool !"           => ~20 bytes

Ensure that you are correctly storing the shortened names and let you Data access framework do the lifting of mapping these to more reasonable names.

Upvotes: 2

Related Questions