CODEWITHSUNDEEP
mongodbleft-joinaggregation-frameworksharding
Map X
Map X

Reputation: 444

MongoDB to Use Sharding with $lookup Aggregation Operator

$lookup is new in MongoDB 3.2. It performs a left outer join to an unsharded collection in the same database to filter in documents from the “joined” collection for processing.

To use $lookup, the from collection cannot be sharded.

On the other hand, sharding is a useful horizontal scaling approach.

What's the best practise to use them together?

Upvotes: 32

Views: 13246

Answers (3)

Dimitar Atanasov
Dimitar Atanasov

Reputation: 169

It's now possible. From the MongoDB documentation:
Starting in MongoDB 5.1, you can use $lookup with sharded collections.

Upvotes: 0

Cuong Bui
Cuong Bui

Reputation: 36

As mentioned in MongoDb document "In the $lookup stage, the from collection cannot be sharded. However, the collection on which you run the aggregate() method can be sharded" https://docs.mongodb.com/manual/reference/operator/aggregation/lookup/

db.shardedCollection.aggregate([
   { $lookup: { from: "unshardedCollection", ... } }
])

This is the best practise to use them together

Upvotes: 0

JohnnyHK
JohnnyHK

Reputation: 312095

As the docs you quote indicate, you can't use $lookup on a sharded collection. So the best practice workaround is to perform the lookup yourself in a separate query.

  1. Perform your aggregate query.
  2. Pull the "localField" values from your query results into an array, possibly using Array#map.
  3. Perform a find query against the "from" collection, using a query like {foreignField: {$in: localFieldArray}}
  4. Merge your results into whatever format you need.

Don't let the $lookup limitation stop you from sharding collections that require it for scalability, just perform the lookup function yourself.

Upvotes: 42

Related Questions