Is a bucket pattern in MongoDb the best way to handle large unbounded arrays?

Question

I'm implementing social features to a MERN stack app (follow/unfollow users), and trying to come up with a good MongoDB solution for avoiding issues with potentially large unbounded arrays of followers. Specifically I'm hoping to avoid:

MongoDB having to move a large follower array on disk and rebuild indexes as it grows larger
hitting the 16mb bson limit if a user ever hits a very large number of followers (> 1 million)
slow performance when querying/returning followers to display via pagination, or when calculating/displaying follower count

From everything Iv'e researched, it seems like using a bucket pattern approach is the best solution... two good articles I found on this: https://www.mongodb.com/blog/post/paging-with-the-bucket-pattern--part-1 https://www.mongodb.com/blog/post/paging-with-the-bucket-pattern--part-2

I've started to approach it like this... Follower model:

const mongoose = require('mongoose');
const Schema = mongoose.Schema;

const FollowerSchema = new Schema({
  user: {
    type: Schema.Types.ObjectId,
    ref: 'user',
  },
  // creating an array of followers
  followers: [
    {
      user: {
        type: Schema.Types.ObjectId,
        ref: 'user',
      },
      datefol: {
        type: Date,
        default: Date.now,
      },
    },
  ],
  count: {
    type: Number,
  },
  createdate: {
    type: Date,
    default: Date.now,
    required: true,
  },
});

module.exports = Follower = mongoose.model('follower', FollowerSchema);

Upsert in Node.js api to add a follower to an array bucket (each bucket will contain 100 followers):

const follow = await Follower.updateOne(
        { user: req.params.id, count: { $lt: 100 } },
        {
          $push: {
            followers: {
              user: req.user.id,
              datefol: Date.now(),
            },
          },
          $inc: { count: 1 },
          $setOnInsert: { user: req.params.id, createdate: Date.now() },
        },
        { upsert: true }
      );

Basically every time a follower is added, this will add them to the first bucket found that contains less than 100 followers (tracked by the count).

Is this the best approach for handling potentially large arrays? My concerns are:

if someone unfollows a user and the app runs a $pull to remove the follower from the array in one of the buckets... multiple buckets could then contain less than 100 followers. New followers will no longer be added to the most recent bucket so later when querying and trying to return followers based on most recent by bucket createdate... some of the newest followers might be in an older bucket and not returned correctly. The articles above mention some expressive update instructions introduced in MongoDb 4.2 that solve this problem, but it's not really clear to me how.
if I corrected for that by returning all follower buckets for a user and sorting by follow date... it seems like that could become very slow if someone had tons of followers
if I want to be able to paginate and return 100 followers per page, starting with the latest, how would that work with this approach? Should I add a pagenumber entry to the model and somehow have it be incremented each time a bucket is created (first bucket contains pagenumber 1, next pagnumber 2 etc), then on the front end if a user jumps to follower page 500 a query runs to pull bucket 500?

Is a bucket pattern in MongoDb the best way to handle large unbounded arrays?

Answers (1)

Related Questions