CODEWITHSUNDEEP
mongodbconcurrencyaggregation-framework
Eric
Eric

Reputation: 1444

MongoDB: Collisions with $out operating on same collection

I am implementing query result caching in my database by issuing collection.aggregate(..) with a query and an $out operator as the last stage. Every so often, there will be two identical queries that attempt to output their cached result at the same time, and they collide with an error message like:

{"message":"$out failed: indexes of target collection dbName.collectionName changed during processing." ... }

The first collection.aggregate(..) is still writing documents into the collection when the second tries to replace the collection to start writing its own documents.

I'm looking for suggestions of how to avoid this problem. I don't know if I can execute $out with some option to have it not replace the collection or if I can have it lock at the collection level rather than yielding in the middle of writing documents, etc.

Any ideas?

Upvotes: 1

Views: 1730

Answers (1)

Eric
Eric

Reputation: 1444

I wasn't able to figure out a way to avoid the $out collisions directly. However, my team member came up with this solution:

  1. Randomize the name of the output collection and then once it's written,
  2. Rename the collection to the desired name.

Here's the code for what I did (using the monk node.js driver for MongoDB):

function cacheResults( queryName, collection, query ) {
  var tempCollectionName = queryName + Math.random();
  var outQuery = query.concat( { $out: tempCollectionName } );
  return collection.aggregate( outQuery )
    .then( () => db.collection( tempCollectionName ) )
    .then( cachedCollection => cachedCollection.executeWhenOpened() )
    .then( nativeCollection => nativeCollection.rename( queryName, { dropTarget: true } ) )
    .then( () => db.collection( queryName ) );

Upvotes: 1

Related Questions