CODEWITHSUNDEEP
mongodbgridfs
remedix
remedix

Reputation: 492

Huge size on mongodb's gridfs. Should I compact?

I've posted this question elsewhere as well with no answers and decided to try here as well. So here it goes:

I'm running mongodb and grid.fs to store small files (less than 20mbs). These are part of a replica set. I currently have more than 350000 files stored.

I've noticed that the chunks collection takes around 700GB of preallocated space where the actual chunks consist of ~40GB. Even though there are 700GB of data preallocated, this keeps expanding over time.

Keep in mind that every 15 minutes or so I delete files older than 5 days. So in theory my fs.chunks and fs.files size should remain around the same over time.

Here's my fs.chunks stats

rs0:PRIMARY> db.fs.chunks.stats()
{
    "ns" : "collection.fs.chunks",
    "count" : 470388,
    "size" : 43295062144,
    "avgObjSize" : 92041.17057407927,
    "storageSize" : 757794040352,
    "numExtents" : 373,
    "nindexes" : 2,
    "lastExtentSize" : 2146426864,
    "paddingFactor" : 1,
    "systemFlags" : 1,
    "userFlags" : 0,
    "totalIndexSize" : 40356736,
    "indexSizes" : {
        "_id_" : 17431232,
        "files_id_1_n_1" : 22925504
    },
    "ok" : 1
}

Is this behaviour normal ? Can I compact (defrag?) the chunks collection or even claim that preallocated space ? If I cannot reclaim that space (which I'm 99.9% sure I can't) is there a way to ensure that the preallocated space will be used eventually rather than keeps expanding ? Thanks!

Upvotes: 1

Views: 2358

Answers (1)

aks
aks

Reputation: 720

You have a couple of options here:

You could run the compact command on a single collection, or one by one in all the collections you want to shrink.

http://www.mongodb.org/display/DOCS/Compact+Command

db.runCommand( { compact : 'mycollectionname' } )

As noted in document, compact does not actually reclaim disk space, it only defragments and rebuilds collection and associated indexes.

Use "--repair" option to validate / rebuild the data files - This is susceptible to data loss if there is any corruption in the database. If you do not have enough space on the same mounted partition, you can use "--repairpath" to specify another location to build the compact files.

For example:

mongod --dbpath /data/db --repair --repairpath /data/db0

Shown here: http://docs.mongodb.org/manual/tutorial/recover-data-following-unexpected-shutdown/

If this is a replica-set another option if the re-sync the node from another replica - This will essentially build the entire database from the other replica node of the replica-set. You can find more details about this at http://docs.mongodb.org/manual/tutorial/resync-replica-set-member/.

Upvotes: 2

Related Questions