CODEWITHSUNDEEP
javascriptandroidmysqlapirest
Harfangk
Harfangk

Reputation: 843

How to ensure data consistency in REST API pagination

I'm building an instant messenger on mobile client that interacts with RESTful API through HTTP requests. The pagination endpoint is quite standard - it has starting location (offset) and number of items in a page (limit). I'm having trouble figuring out how to ensure 100% data consistency with pagination when the database can rapidly change.

For example, with some dozen participants, there could be a dozen new messages in a conversation within a second. I don't think it's far-fetched to guess that some of those messages can alter the database within the time the HTTP request for pagination comes back from the server. Fortunately, since this is a messenger I do not have to consider the possibility of data deletion and consider only the data addition.

Among my research, following two links were quite helpful but didn't provide clear solution:

How to ensure data integrity in paginated REST API?

How to implement robust pagination with a RESTful API when the resultset can change?

The only potential solution I can come up with is using the timestamp of the last object in the previously fetched page. So the HTTP query would have timestamp as a parameter, and the server would return a page of objects created after that timestamp.

Is there any potential problem I'm not seeing, or even better, a much better solution to this issue?

Upvotes: 5

Views: 3911

Answers (1)

Harfangk
Harfangk

Reputation: 843

It seems that the method I've thought of has a name - cursor based pagination.

The link below has a great graphical description and explanation, plus an example in php.

http://www.sitepoint.com/paginating-real-time-data-cursor-based-pagination/

There's also a helpful guide from Django Framework that compares two different pagination techniques (LimitOffsetPagination and CursorPagination).

http://www.django-rest-framework.org/api-guide/pagination/

Cursor based pagination requires a unique, unchanging ordering of items. Facebook and Twitter use some generated IDs. As for me, I've decided to simply use timestamp at object creation, as it supports up to milliseconds precision. That should be good enough for now.

Upvotes: 2

Related Questions