mongoDB: group by failing while querying from pymongo

Question

Here is what I am doing:

>>> import pymongo
>>> con = pymongo.Connection('localhost',12345)
>>> db = con['staging']
>>> coll = db['contract']
>>> result = coll.group(['asset_id'], None, {'list': []}, 'function(obj, prev) {prev.list.push(obj)}') 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "build/bdist.macosx-10.3-fat/egg/pymongo/collection.py", line 908, in group
  File "build/bdist.macosx-10.3-fat/egg/pymongo/database.py", line 340, in command
  File "build/bdist.macosx-10.3-fat/egg/pymongo/helpers.py", line 126, in _check_command_response
pymongo.errors.OperationFailure: command SON([('group', {'$reduce': Code('function(obj, prev) {prev.list.push(obj)}', {}), 'ns': u'contract', 'cond': None, 'key': {'asset_id': 1}, 'initial': {'list': []}})]) failed: exception: BufBuilder grow() > 64MB

and what I see on mongod logs is following

Wed Nov 16 16:05:55 [conn209] Assertion: 13548:BufBuilder grow() > 64MB                                                                           0x10008de9b 0x100008d89 0x100151e72 0x100152712 0x100151954 0x100152712 0x100151954 0x100152712 0x100152e7b 0x100152f0c 0x10013b1d9 0x1003706bf 0x10037204c 0x10034c4d6 0x10034d877 0x100180cc4 0x100184649 0x1002b9e89 0x1002c3f18 0x100433888                                                      0   mongod                              0x000000010008de9b _ZN5mongo11msgassertedEiPKc + 315                                                      1   mongod                              0x0000000100008d89 _ZN5mongo10BufBuilder15grow_reallocateEv + 73
 2   mongod                              0x0000000100151e72 _ZN5mongo9Convertor6appendERNS_14BSONObjBuilderESslNS_8BSONTypeERKNS_13TraverseStackE + 2962
 3   mongod                              0x0000000100152712 _ZN5mongo9Convertor8toObjectEP8JSObjectRKNS_13TraverseStackE + 1682
 4   mongod                              0x0000000100151954 _ZN5mongo9Convertor6appendERNS_14BSONObjBuilderESslNS_8BSONTypeERKNS_13TraverseStackE + 1652
 5   mongod                              0x0000000100152712 _ZN5mongo9Convertor8toObjectEP8JSObjectRKNS_13TraverseStackE + 1682
 6   mongod                              0x0000000100151954 _ZN5mongo9Convertor6appendERNS_14BSONObjBuilderESslNS_8BSONTypeERKNS_13TraverseStackE + 1652
 7   mongod                              0x0000000100152712 _ZN5mongo9Convertor8toObjectEP8JSObjectRKNS_13TraverseStackE + 1682
 8   mongod                              0x0000000100152e7b _ZN5mongo9Convertor8toObjectEl + 139
 9   mongod                              0x0000000100152f0c _ZN5mongo7SMScope9getObjectEPKc + 92
 10  mongod                              0x000000010013b1d9 _ZN5mongo11PooledScope9getObjectEPKc + 25
 11  mongod                              0x00000001003706bf _ZN5mongo12GroupCommand5groupESsRKSsRKNS_7BSONObjES3_SsSsPKcS3_SsRSsRNS_14BSONObjBuilderE + 3551
 12  mongod                              0x000000010037204c _ZN5mongo12GroupCommand3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb + 3676
 13  mongod                              0x000000010034c4d6 _ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb + 1350
 14  mongod                              0x000000010034d877 _ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_10BufBuilderERNS_14BSONObjBuilderEbi + 2151
 15  mongod                              0x0000000100180cc4 _ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_10BufBuilderERNS_14BSONObjBuilderEbi + 52
 16  mongod                              0x0000000100184649 _ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_ + 10585
 17  mongod                              0x00000001002b9e89 _ZN5mongo13receivedQueryERNS_6ClientERNS_10DbResponseERNS_7MessageE + 569
 18  mongod                              0x00000001002c3f18 _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE + 1528
 19  mongod                              0x0000000100433888 _ZN5mongo10connThreadEPNS_13MessagingPortE + 616
Wed Nov 16 16:05:55 [conn209] query staging.$cmd ntoreturn:1 command: { group: { $reduce: CodeWScope( function(obj, prev) {prev.list.push(obj)}, {}), ns: "contract", cond: null, key: { asset_id: 1 }, initial: { list: {} } } } reslen:111 1006ms

I am very new to both pymongo and Mongodb, and dont't know how to resolve this, please help

Thank you

Answer 1

The relevant part of your stacktrace is:

exception: BufBuilder grow() > 64MB

Basically, Mongo doesn't allow you to have any document greater than 64MB. See this question for more details (the size limit has been bumped to 64MB since then.)

I'm not sure what you're trying to do with that query. It sort of looks like you want to get a list of objects for each asset_id. However, your collection is going to grow past capacity because you're never differentiating between objects in your group. Try setting your initial to {'asset_id': '', 'objects': []} and reduce to function(obj, prev) {prev.asset_id = obj.asset_id; prev.objects.push(obj) although there are much more efficient ways of doing this query.

Alternatively, if you're trying to get all the documents matching an ID, try:

coll.find({'asset_id': whatevs})

If you're trying to get a count of the objects, try this instead:

coll.group(
    ['asset_id'], None, {'asset_id': '', 'count': 0}, 
    'function(obj, prev) {prev.asset_id = obj.asset_id; prev.count += obj.count}'
)