Web Services with large volume of data

Question

I am currently designing a set of Web Services using JAX-WS and the bottom-up approach.

The problem with some of my services is that they will be used for reporting activities, and therefore they are very likely to return large volumes of data. My biggest concern is obviously the response time from a client's perspective.

I am looking at different options:

• Pagination: this involves modifying my operations and making them aware of the large volume of data returned by adding one or several parameters in order to select the page or chunk of data to be retrieved.

• Data filtering: this one also involves altering my service by adding an operation parameter to filter the data by some criteria.

After doing a lot of research, it looks weird to me that there are not best practices in this area, as I pressume I am not the first one to come across this problem.

Can anyone provide some advice or point out resources where I can find information on this issue?

Thanks in advance.

Answer 1

Pagination all the way, but really, I would use both. If you are dealing with massive amounts of data coming back, your users almost certainly want to be able to filter it. Looking at thousands of records at once isn't useful to anyone. Any time you are in danger of getting a large amount of data back from anything, paging is the way to go.

I would say paging is the best practice here.

Answer 2

You probably want both.

If you can pass filter arguments to the webservice, do it, and apply them in the webservice instead of sending all the data and filtering on the client.

If the client displays data page-wise, then provide a pagination mechanism in the webservice. Make it optional if other processes may need to get the full dataset at once and can afford to wait a little longer.