Design pattern for real time data

Question

I have a stream of real-time events coming in. I am developing a front-end to display this stream to a user so that:

• User will be able to search using one of the attributes present in the data
• User will be able to sort the data

Each event in the stream has a state associated with it which can be changed at any point so I keep querying the server for the last x minutes worth of data and change the state accordingly on the client-side.

My first attempt was using pagination. However, this has been giving rise to the following problems:

• When the user is viewing say, the 15th page and a new event gets added, I need to remember the current page the user is at and make sure that does not change. Even if the newly added data does not push things too much, it will affect the position of the event in the list and hence the user has to search where the previous event has moved to.
• When the user sorts by some particular attribute, I need to re-sort the incoming data and display it. I still face the same problem as above.

Is there a design pattern for this interaction? Twitter uses infinite scrolling. This pattern is awesome but I am not quite sure how to adapt it to situations:

• Where the user can sort the data.
• Where I am using a back-end database to provide data to the front-end. How should the pagination queries be designed?

Any suggestions?

Answer 1

It sounds like shifting sorting to the front-end is the best way to achieve your desired result. I won't go deep into infinite scrolling/pagination because there are probably better resources out there, but basically keep track of the last row (or page) retrieved and send that along with your request. Hopefully this helps with the other issues blocking you from implementing that.

Aside: Depending on how real-time the data you should probably look into other methods of obtaining it like long polling or WebSockets instead of repeated calls every X minutes.

Response Format

{
    'status' : 'OK',
    'last_row_retrieved': 20,
    'events' : [{
        'html' : "<div>The regular response</div>",
        'sort_attr_1' : 10,
        ...
    }]
}

Front-End

//Maintain a container for the results you get each call.
var myevents = [];
$.post(...,function(response) {
    var data = $.parseJSON(response);
    var events = data.events;
    for(var i=0;i<events.length;i++) {
        //Save each event
        myevents.push(events[i]);
    }
    DisplayEvents();
});
//When user tries to sort do so with custom sorting function
myevents.sort(function(a,b) { return a.sort_attr_1 - b.sort_attr_2;});
DisplayEvents();

function DisplayEvents() {
    //loop through myevents and display them however you're doing it
}