CODEWITHSUNDEEP
mongodbaggregation-framework
ranjjose
ranjjose

Reputation: 2148

mongodb 2.4.1 failing with 'process out of memory'

I have the following mongo versions

db version v2.4.1    
MongoDB shell version: 2.4.1,

and

db version v2.2.1-rc1, pdfile version 4.5, 
MongoDB shell version: 2.2.1-rc1

installed on 64-bit windows 7 machine.

I have a collection having 10001000 (10 million+) records, when I use V 2.4.1 to aggregate, it fails with the following error:

Fatal error in CALL_AND_RETRY_2
Allocation failed - process out of memory

However when I use V 2.2.1-rc1, to aggregate the same collection, it works fine and gives result in around 1 minute.

Sample document of the collection that is being aggregated:

{

    "_id" : ObjectId("516bdd1c39b10c722792e007"),
    "f1" : 10000010,
    "f2" : 10000000,
    "key" : 0
}

Aggregation Command:

{$group: {"_id": "$key", total: {$sum: "$f1"}}}

Command used to populate records:

for(var i = 10011000; i < 10041000; ++i)
{ 
    db.testp.insert({"f1": i+10, "f2": i, "key": i%1000})
}

Upvotes: 3

Views: 1318

Answers (1)

Ross
Ross

Reputation: 18111

How much memory do you have? Could it be the $group is taking up more than 10% of available memory and causing the error? See the aggregation documentation on memory for cumulative operators.

edit 1:

Out of interest - does the aggregation work outside the shell? eg calling it from a driver. I have seen similar v8 errors and as the shell was updated to v8 in 2.4 Theres a chance it could be that.

edit 2:

If the resulting array is too big in the shell then that can also trigger the error: see SERVER-8859. To work around you might need to run multiple aggregations, either by doing a $match early on to limit the working set or even a $skip and $limit to paginate through the result set.

I tried your aggregation with 10,070,999 docs on 2.4.1 on a mac and didn't get the error

Upvotes: 4

Related Questions