KeystoneJS list pagination of aggregate results

Question

In KeystoneJS we are currently using aggregation to query Mongo and just grabbing the whole set and returning that. Now, that we're getting more data, we need to paginate. KeystoneJS supports pagination, but from the looks of it is only supporting the find command in the where option parameter. Is there any way that the pagination provided by KeystoneJS can paginate the results from an aggregation query?

Example aggregate query:

keystone.list('Posts').model.aggregate([
{'$match': {"state":"published"}},
{'$unwind': {'path':"$actions", 'preserveNullAndEmptyArrays': false}},
      {'$lookup': {
        'from':"actions",
        'localField': "actions",
        'foreignField': "_id",
        'as': "actions"}},
      {'$match': {"actions.key": action}},
{'$unwind': {'path':"$postTopics"}},
      {'$lookup': {
        'from':"posttopics",
        'localField': "postTopics",
        'foreignField': "_id",
        'as': "posttopics"}},
      {'$match': {"posttopics.key": topic}},
{'$unwind': {'path':"$postSubTopics"}},
      {'$lookup': {
        'from':"postsubtopics",
        'localField': "postSubTopics",
        'foreignField': "_id",
        'as': "postsubtopics"}},
      {'$match': {"postsubtopics.key": subtopic}},
{'$unwind':
    {'path':"$postSubTopics",
     'preserveNullAndEmptyArrays': true}},
  {'$unwind':
    {'path':"$postTopics",
     'preserveNullAndEmptyArrays': true}},
  {'$lookup': {
    'from':"postsubtopics",
    'localField': "postSubTopics",
    'foreignField': "_id",
    'as': "postsubtopics"}},
  {'$lookup': {
    'from':"posttopics",
    'localField': "postTopics",
    'foreignField': "_id",
    'as': "posttopics"}},
  {'$match': {
    '$or':
      [
        { "postsubtopics.searchKeywords": keyword },
        { "posttopics.searchKeywords": keyword }
      ]
  }}
]).sort('-publishedDate');

The results that return back from this I would like to be able to paginate through. I am exploring using mongoose to do it or just filtering through the array with javascript, but since I see that Keystone has pagination built in, I wanted to ask the contributors if this is supported.

Answer 1

For those who find this later, I was able to leverage Mongoose to help clean up this query. Unfortunately, with paginate it doesn't seem possible yet to take into consideration post populate filtering. The paginate will return all posts that match the where clause, but this doesn't help if I need to filter more on populated fields. By the time we populate, the pagination has already been done. So, if paginate returns 10 out of 30 initially, after I do a filter in the exec I may only end up with 3 out of the 10. I ended up creating a global PostsPaginator object to help me with this instead of using Keystone's paginate.

var populateObjects = [
      {
        path: 'actions',
        match: { 'key': action}
      },
      {
        path: 'postTopics',
        match: { '$or': [
          {'key': topic},
          { "searchKeywords": keyword }]
        }
      },
      {
        path: 'postSubTopics',
        match: { '$or': [
          {'key': subtopic},
          { "searchKeywords": keyword }]
        }
      }
    ];

keystone.list('Posts').model
.find({'state': 'published',
           'publishedDate': {'$lte': currentDate}})
.populate(populateObjects)
.sort({'publishedDate': -1}).lean()
.exec(function (err, result) {
 result = result.filter((doc) => {//logic to check if wanted fields are populated}
}



// In another file...

    PostsPaginator.getPostsFromPostsArray = function(req, res, postsArray, pageNumber, postsPerPage) {
      /*attributes:
          pageNumber  - number:   the current page we are displaying for
          totalPosts  - number:   total number of posts that exist for the query
          totalPages  - number:   total number of pages there will be for pagination
          pagePosts   - array:    the posts that will be returned to display for the page
          isValidPage - boolean:  if invalid page number is input return false
          pagePath    - string:   the url path of the page requested
           */
      var posts = {};
      postsPerPage = postsPerPage || 10;
      posts.pageNumber = pageNumber;

      posts.totalPosts = postsArray.length;
      posts.totalPages = Math.ceil(postsArray.length / postsPerPage);

      var start = postsPerPage * (pageNumber - 1);
      var end = postsPerPage * pageNumber;
      posts.pagePosts = postsArray.slice(start, end);
      posts.isValidPage = start <= posts.totalPosts;
      posts.pagePath = req.path;
      return posts;
    };

Answer 2

Firstly, have you tried simply:

keystone.list('Posts').model.paginate({
                page: req.query.page || 1,
                perPage: 10,
                maxPages: 10,
            })
            .aggregate([...])

I don't really see a reason as to why that shouldn't work, although I'm not familiar with how mongoose works under the hood. Keystone certainly allows you to make a pagination call like that though.

Failing that, you could split this up into two calls. First do a standard pagination call on the Posts, to get the ids of the posts on that page and save that to an array. You could then do an aggregate call which first matches the posts which have ids in the array you just created. There's a StackOverflow post on how to match if a value is in an array here. That should achieve what you want?

Finally, are you sure all that aggregation etc is necessary. I'm not 100% on what is being achieved here as I've never actually used aggregate, however you could have the postTopics as a relationship field on the post, which would allow you to call populate instead? Similarly, each topic could have a relationship to its subtopics etc etc. I assume your use case is too complicated for this, but thought I'd mention it just in case.