MongoDB: How to combine array elements from multiple documents?

Question

I am storing sensor data collected from mobile devices in mongodb using size-based bucketing approach. Since mongodb has 16mb limit on document size, my data is spread across multiple documents like this.

Document 1:

{
    "_id" : ObjectId("5ca411ad4fa0fa4c4893b025"),
    "experimentId" : ObjectId("5ca411ac4fa0fa4c4893b024"),
    "collectedData" : 
            [{
                    "pressure" : 125,
                    "lat" : 37.5
            },
            {
                    "pressure" : 155,
                    "lat" : 47.5
            },
            {
                    "pressure" : 128,
                    "lat" : 36.5,
            }]
}

Document 2:

{
    "_id" : ObjectId("5cace327b322a8893df703b3"),
    "experimentId" : ObjectId("5ca411ac4fa0fa4c4893b024"),
    "collectedData" : [

            {
                    "pressure" : 124,
                    "lat" : 35.5
            },
            {
                    "pressure" : 115,
                    "lat" : 42.5
            },
            {
                    "pressure" : 15,
                    "lat" : 52.5
            }
    ]
}

Notice that documents created for same experiment/task have same "experimentId".

How do I combine "collectedData" from multiple documents with same "experimentId" so that I can use this information for data visualization later?

I expect the output to be in below format.

{
    "collectedData" : [
            {
                    "pressure" : 125,
                    "lat" : 37.5
            },
            {
                    "pressure" : 155,
                    "lat" : 47.5
            },
            {
                    "pressure" : 128,
                    "lat" : 36.5,
            },
            {
                    "pressure" : 124,
                    "lat" : 35.5
            },
            {
                    "pressure" : 115,
                    "lat" : 42.5
            },
            {
                    "pressure" : 15,
                    "lat" : 52.5
            }

    ]
}

Answer 1

You may use aggregation framework to collect all scattered arrays. However, you might still face 16 MB limit restriction based on how much data you are projecting after aggregation.

db.getCollection("your_collection_name").aggregate(
    [
        { 
            "$unwind" : {
                "path" : "$collectedData"
            }
        }, 
        { 
            "$group" : {
                "_id" : "$experimentId", 
                "collectedData" : {
                    "$addToSet" : "$collectedData"
                }
            }
        }
    ], 
    { 
        "allowDiskUse" : true
    }
);

You might want to do it in application code for fool-proof solution.