How to group two collections in one query with MongoDB or Mongoose

Question

I have page like this. And the data is from my MongoDB server.

At first, I create User collection and put those user data. But I add the Counter collection because, I want to show their View Count.

Each user has multiple pages, and the View Count data is the sum of the Counter data.

User collection's are look like this and I only need login, id(unique) data.

Users Collection

{
    "_id": {
        "$oid": "5f7e92b88dc8f64cb4e6c2c0"
    },
    "login": "mojombo",
    "id": 1,
    "avatar_url": "https://avatars0.githubusercontent.com/u/1?v=4",
    "url": "https://api.github.com/users/mojombo",
    "html_url": "https://github.com/mojombo",
    "type": "User",
    "site_admin": "false",
    "name": "Tom Preston-Werner",
    "company": null,
    "blog": "http://tom.preston-werner.com",
    "location": "San Francisco",
    "email": "tom@mojombo.com",
    "hireable": null,
    "bio": null,
    "created_at": {
        "$date": "2007-10-20T05:24:19.000Z"
    },
    "updated_at": {
        "$date": "2020-09-22T15:50:44.000Z"
    },
    "registerDate": {
        "$date": "2020-10-08T04:16:56.459Z"
    },
    "__v": 0
}

Counters collection

{
    "_id": {
        "$oid": "5f8e5bde9054ba2477dc2c57"
    },
    "repoName": "ale",
    "repoNumber": 171780764,
    "userName": "technicalpickles",
    "userNumber": 159,
    "viewDate": "2020-10-20",
    "count": 1
}

In Counters collection I only need userName, count field to calculate counter data.

So I try for loop and aggregate method but the problem is User data is too much so I have to send too many queries to server. (If I have 10000 users data I have to send 10000 requests) So, I must not to use for loop method.

router.get(`/sitemap/0`, async (req, res, next) => {
  let name;
  let dataArray = [];
  let pageNumber = (Number(req.params.page) * 10000); // Current Page Number
  let nextPage = (Number(req.params.page) + 1) * 10000; // Next Page Number
  let pageResult; // Page Result
  try {
    let users = await User.find({}, 'login id').limit(1000)
    for (let i = 0; i < 1000; i++) {
      name = users[i].login
      let counters = await Counter.aggregate([{
          $match: {
            id: users.id,
            userName: users[i].login
          }
        },
        {
          $group: {
            _id: `${users[i].login}`,
            count: {
              $sum: "$count"
            }
          }
        }
      ])
      dataArray.push(counters)
      console.log(counters)
    }
    console.log(dataArray)

  } catch (e) {
    // throw an error
    throw e;
  }
  res.render("sitemap", {
    dataArray
  })
});

Codes result

So I want to send single query and I think it should be use 'aggregate' method. But without 'for loop'.

I want to join Users collection and Counters collection but I heard that in MongoDB & Mongoose there are no join function.

I just want to make the result like this. Is there anyway to make it like this?

{
    "_id": {
        "$oid": "5f7e92b88dc8f64cb4e6c2c0"
    },
    "login": "mojombo", --------------Match login with counter collection 'userName' value
    "id": 1,  ------------------------Match id with counter collection 'id' value
    "count": [Sum of counters count data and it should be Number] ------- Only this field is added
    "avatar_url": "https://avatars0.githubusercontent.com/u/1?v=4",
    "url": "https://api.github.com/users/mojombo",
    "html_url": "https://github.com/mojombo",
    "type": "User",
    "site_admin": "false",
    "name": "Tom Preston-Werner",
    "company": null,
    "blog": "http://tom.preston-werner.com",
    "location": "San Francisco",
    "email": "tom@mojombo.com",
    "hireable": null,
    "bio": null,
    "created_at": {
        "$date": "2007-10-20T05:24:19.000Z"
    },
    "updated_at": {
        "$date": "2020-09-22T15:50:44.000Z"
    },
    "registerDate": {
        "$date": "2020-10-08T04:16:56.459Z"
    },
    "__v": 0
}

Answer 1

• $lookup with counters collection using pipeline stage, let to pass 2 fields id, login to match with counters collection
• $match both fields conditions
• $addFields to count sum of count field from counters array, using $reduce to iterate loop and $add to sum values of count field
• $skip for pagination
• $limit pass your limit of documents

let page = 1; // start pagination from first
let limit = 1000;
let skip = (page - 1) * limit;
let users = await User.aggregate([
  {
    $lookup: {
      from: "counters",
      let: {
        id: "$id",
        login: "$login"
      },
      pipeline: [
        {
          $match: {
            $expr: {
              $eq: ["$userName", "$$login"],
              $eq: ["$userNumber", "$$id"]
            }
          }
        }
      ],
      as: "count"
    }
  },
  {
    $addFields: {
      count: {
        $reduce: {
          input: "$count",
          initialValue: 0,
          in: { $add: ["$$value", "$$this.count"] }
        }
      }
    }
  },
  { $skip: skip },
  { $limit: limit }
])

Playground