CODEWITHSUNDEEP
mongodb
Dave Edelhart
Dave Edelhart

Reputation: 1101

Mongo is taking a HUGE amount of space to save this data?

https://gist.github.com/1173528#comments

shows the structure of the data file ...

the short version is 

{ "img_ref" : {
  "$ref" : "mapimage",
  "$id" : ObjectId("4e454599f404e8d51c000002")
},
  "scale" : 128, "image" : "4e454599f404e8d51c000002", "tile_i" : 0, "tile_j" : 9, "w" : 9, "e" : 10, "n" : 0, "s" : 0,
  "heights" : [
    [
        0,
        2,
        0,
        1,
        515,
        0,
        256,
        ...], [...]
, _id: ObjectId("...") }

The stats() on this collection is: 

{
"ns" : "ac2.mapimage_tile",
"count" : 18443,
"size" : 99513670744,
"avgObjSize" : 5395742.056281516,
"storageSize" : 100336473712,
"numExtents" : 74,
"nindexes" : 4,
"lastExtentSize" : 2146426864,
"paddingFactor" : 1,
"flags" : 0,
"totalIndexSize" : 5832704,
"indexSizes" : {
    "_id_" : 786432,
    "img_ref_1_tile_i_1_tile_j_1" : 2236416,
    "image_1" : 1212416,
    "image_1_tile_i_1_tile_j_1_scale_1" : 1597440
},
"ok" : 1
}

Note the average object size, 5,395,742 bytes - or 5 MB! 5 MB for storing 16,384 ints seems pretty extreme!

Upvotes: 1

Views: 311

Answers (2)

mstearn
mstearn

Reputation: 4276

See http://bsonspec.org/#/specification for how things get serialized in mongodb. Arrays are actually very space inefficient especially because we store the index number as a string key for each element. This is less of a problem for small arrays of large elements like strings or objects, but for large arrays of 32-bit ints it is very expensive.

Upvotes: 2

Chris Shain
Chris Shain

Reputation: 51329

MongoDB pre-allocates space for it's databases: http://www.mongodb.org/display/DOCS/Developer+FAQ#DeveloperFAQ-Whyaremydatafilessolarge%3F

What you are likely seeing is that pre-allocation- if you add further items, you probably will not see a further increase in space usage for a long while.

Also: http://www.mongodb.org/display/DOCS/Excessive+Disk+Space

Upvotes: 1

Related Questions