What is a good Practice for Pagination using Page Number & Limit in Go?

Question

I have created Pagination in Go using Page Number & Limit. Where Limit & Page Number are INT

and I created pagination like :

MONGO_SESSION.Find(nil).Skip(pageNumber*limit).Limit(limit).Sort("_id").All(&RETURN_STRUCT)

It's working fine. But when I send page number or limit ZERO. By default mongo DB return all records because nothing to skip and limit.

So my question is that what is good practice in case of zero limit and zero page number.

Practice 1: Send all data. Don't send error response.

Practice 2: Send error response saying "Page number and limit can't be zero"

Note: I can't hardcode limit or page number.

Any suggestion would be appreciated.

Answer 1

This question is somewhat opinion-based (and therefore somewhat off-topic for StackOverflow), but I think some advice or general practice may be helpful and useful for others. So therefore the following of the answer is my opinion.

You as the developer of the server application are responsible for the safety and security of the server, and for the utilization of the server's resources. You as the developer should trust clients to the least extent necessary.

That said, sending all documents when client fails to specify the limit (either accidentally or purposefully) is the worst way to handle the situation. It's like screaming out: "Hey, clients and hackers, here's an endpoint, and if you want to DoS attack my server, just call this endpoint a couple of times".

To protect your server, you should have a safety limit even for the "limit" parameter, because allowing any value for it may be just as bad: just because you force clients to specify a limit doesn't protect your server, "bad" clients may just as well send a limit like 1e9 which will most likely include all your documents.

My advice is to always have meaningful defaults and safety limits. Default values should always be documented, safety limits are not so important (but could be documented as well).

So how you should handle it is:

1. If limit is missing, apply the defaults. If you can't have defaults, just skip this step (although there must be a good reason not to have / allow a default).

2. Limit should be checked against a safety limit. If it exceeds the safety value, use the safety limit (max allowed limit).

3. If the server has the "right" to alter the requested limit (e.g. using a default when limit is missing, or capping the limit based on a safety limit), the server should communicate back to the client the limit that was actually used to serve the request.

And regarding efficient MongoDB paging: I only recommend using Query.Skip() and Query.Limit() for "small" document counts. For an efficient paging that "scales" with the number of documents, check out this question+answer: Efficient paging in MongoDB using mgo

Answer 2

I believe, that paging should be used only for cases when the size of the collection is big (otherwise just show all the data at once and don't fiddle with paging at all).

However, if the collection is reasonably big, then sending all the data is a bad idea.

There is an additional issue with "skip" statement (it's not unique to mongo though): in order to skip N records, the db has to do a full scan (full collection scan in case of mongo), so it will take more time to fetch results for page N + 1 than for page N.

Now, in order to deal with it, there is a "trick": Don't work with skip at all, instead "memorize" the last document id (its already indexed anyway and you sort by _id anyway). Then the queries will be (pseudocode since I don't speak 'Go'):

• For the first query: Find().sort(_id).limit(limitSize)

• For subsequent queries: Find ().where(_id > lastMemorizedId).sort(_id).limit(limitSize)

Answer 3

I also faced the same problem. I preferred to send Error Response instead of showing all the data.

Because It's a heavy transaction for DB if DB has to send all data. On small collections, it's work fine but for the large collections, it's gonna hang DB.

So send an error response.