Insert an array of documents into a model

Question

Here's the relevant code:

var Results = mongoose.model('Results', resultsSchema);
var results_array = [];

_.each(matches, function(match) {
    var results = new Results({
        id: match.match_id,
        ... // more attributes
    });
    results_array.push(results);
});
callback(results_array);
});
   }
], function(results_array) {
   results_array.insert(function(err) {
       // error handling

Naturally, I get a No method found for the results_array. However I'm not sure what else to call the method on.

In other functions I'm passing through the equivalent of the results variable here, which is a mongoose object and has the insert method available.

How can I insert an array of documents here?

** Edit **

function(results_array) {
            async.eachLimit(results_array, 20, function(result, callback) {
                result.save(function(err) {
                    callback(err);
                });
            }, function(err) {
                if (err) {
                    if (err.code == 11000) {
                        return res.status(409);
                    }
                    return next(err);
                }
                res.status(200).end();
                });
            });

So what's happening:

When I clear the collection, this works fine. However when I resend this request I never get a response.

This is happening because I have my schema to not allow duplicates that are coming in from the JSON response. So when I resend the request, it gets the same data as the first request, and thus responds with an error. This is what I believe status code 409 deals with.

Is there a typo somewhere in my implementation?

Edit 2

Error code coming out:

{ [MongoError: insertDocument :: caused by :: 11000 E11000 duplicate key error index:       
test.results.$_id_  dup key: { : 1931559 }]
    name: 'MongoError',
    code: 11000,
    err: 'insertDocument :: caused by :: 11000 E11000 duplicate key error index:    
test.results.$_id_  dup key: { : 1931559 }' }

So this is as expected. Mongo is responding with a 11000 error, complaining that this is a duplicate key.

Edit 3

if (err.code == 11000) {
    return res.status(409).end();
}

This seems to have fixed the problem. Is this a band-aid fix though?

Answer 1

You seem to be trying to insert various documents at once here. So you actually have a few options.

Firstly, there is no .insert() method in mongoose as this is replaced with other wrappers such as .save() and .create(). The most basic process here is to just call "save" on each document you have just created. Also employing the async library here to implement some flow control so everything just doesn't queue up:

async.eachLimit(results_array,20,function(result,callback) {
    result.save(function(err) {
        callback(err)
    });
},function(err) {
   // process when complete or on error
});

Another thing here is that .create() can just take a list of objects as it's arguments and simply inserts each one as the document is created:

Results.create(results_array,function(err) {

});

That would actually be with "raw" objects though as they are essentially all cast as a mongooose document first. You can ask for the documents back as additional arguments in the callback signature, but constructing that is likely overkill.

Either way those shake, the "async" form will process those in parallel and the "create" form will be in sequence, but they are both effectively issuing one "insert" to the database for each document that is created.

For true Bulk functionality you presently need to address the underlying driver methods, and the best place is with the Bulk Operations API:

mongoose.connection.on("open",function(err,conn) {

    var bulk = Results.collection.initializeUnorderedBulkOp();
    var count = 0;

    async.eachSeries(results_array,function(result,callback) {
        bulk.insert(result);
        count++;

        if ( count % 1000 == 0 ) {
            bulk.execute(function(err,response) {
               // maybe check response
               bulk = Results.collection.initializeUnorderedBulkOp();
               callback(err);
            });
        } else {
            callback();
        }

    },function(err) {
        // called when done
        // Check if there are still writes queued
        if ( count % 1000 != 0 )
            bulk.execute(function(err,response) {
               // maybe check response
            });
    });

});

Again the array here is raw objects rather than those cast as a mongoose document. There is no validation or other mongoose schema logic implemented here as this is just a basic driver method and does not know about such things.

While the array is processed in series, the above shows that a write operation will only actually be sent to the server once every 1000 entries processed or when the end is reached. So this truly does send everything to the server at once.

Unordered operations means that the err would normally not be set but rather the "response" document would contain any errors that might have occurred. If you want this to fail on the first error then it would be .initializeOrderedBulkOp() instead.

The care to take here is that you must be sure a connection is open before accessing these methods in this way. Mongoose looks after the connection with it's own methods so where a method such as .save() is reached in your code before the actual connection is made to the database it is "queued" in a sense awaiting this event.

So either make sure that some other "mongoose" operation has completed first or otherwise ensure that your application logic works within such a case where the connection is sure to be made. Simulated in this example by placing within the "connection open" event.

It depends on what you really want to do. Each case has it's uses, with of course the last being the fastest possible way to do this as there are limited "write" and "return result" conversations going back and forth with the server.