CODEWITHSUNDEEP
mongodbindexing
dunn less
dunn less

Reputation: 623

mongodb memory confusion. page fault and index size

I'm utterly confused on how much actual memory is being used by my mongodb..

issuing db.stats(1024*1024) i get an output 

> db.stats(1024*1024)
{
    "db" : "test",
    "collections" : 9,
    "objects" : 38035503,
    "avgObjSize" : 535.2138471259339,
    "dataSize" : 19414,
    "storageSize" : 22280,
    "numExtents" : 62,
    "indexes" : 12,
    "indexSize" : 4039,
    "fileSize" : 30642,
    "nsSizeMB" : 16,
    "ok" : 1
}

So i have a little less then 40 million objects and an index size of 4GB. Running this on a windows server with 17GB RAM, looking at the Resource Monitor and task manager, my mongodb instance have a working set of 17GB and a Shareable of the same amount.

My first though was that "ok so it has pre-allocated datafiles so that there will be faster inserts when it actually needs the space".

But then i started mongostat and could clearly see a very much to high page fault:

insert  query update delete getmore command flushes mapped  vsize    res faults
 time
   220   2419      0      0       0     221       0  27.9g  56.1g  16.8g   2659

So im guessing that it actually has run out of memory and is now swapping and picking up directly from disk.

My question basically is how can i actually determine the amount of data that i will be able to store before i run out of RAM?. The stats tell me that it only has 4GB worth of index, but mongo seems to be choked,swapping and consuming all 17GB. What am i missing?

Upvotes: 0

Views: 779

Answers (1)

Asya Kamsky
Asya Kamsky

Reputation: 42342

You have 17GB of RAM available for about 20GB of data plus 4GB of indexes (and that's just on this DB - I don't know if there is more in your system, but mongostat says you have total datafile size of about 28GB).

If you are querying over your entire dataset, wouldn't you expect that eventually new data records being read can no longer fit into RAM without displaying some of the previously read (and loaded into RAM) records? That's what you're seeing with page faulting.

It's not data storage that you are concerned about, it's data being used or queries or accessed. If it all can't stay in RAM then it will get swapped out and then will need to be swapped back in when accessed again.

Upvotes: 2

Related Questions