Retrieve huge data from rest api

Question

I am calling third party rest api.this rest api returns more then 2 thousand records that takes much time to retrieve this information.I want to show these records in asp.net page. Records display in page successfully but problem is its takes much time to display/retrieve this information.

I call this api against dropdown change so api call might be again and again.means it is not one time call and also i cann't cache this information because in server end may be information will change.

So How I reduce download time or any trick(paging is not supported),Any suggestion?

Thanks in advance

Answer 1

Well there's always a workaround for any given problem even when it doesn't come to your mind at first sight.

I had one situation like this, and this is what we did:

• First, keep in mind that real-time consistency is just something that cannot be achieved without writing extensive code. For example, if a user requests data from your service and then that data is rendered on the page, potentially the user could be seeing out-of-data data if at that precise time the source is updated.

  So we need to accept the data will be consistent eventually

• Then consider the following, replicate the data in your database using an external service periodically

• Expose this local data allowing paging and consume it from your client code

As any design decision, there are trade-offs that you should evaluate to take your final choice, these were problems we had with a design like this:

• Potentially phe data will not be consistent when the user requests it. You can poll the data to the server every xxx minutes depending on how often it changes (Note: usually we as developers, we are not happy with this kind of data inconsistency, we would like to see everything on real-time, but the truth, is that the end users, most of the time can live with this kind of latency because that's the way they use to work in real-life, consult with your end-users to gather their point of views about this. You can read more about this from Eric Evans in the Blue Book)

• You have to write extra code to maintain the data locally

Sadly when you are dealing with a third party service with these conditions, there are few options to choose from, I recommend you to evaluate them all with your end-users explaining their trade-offs to choose the solution that best fits your needs

Alternatives:

• Since I already said the data will not be consistency all the time, then perhaps you could consider caching the call to the service =)

• Usually the easiest way is always the best way to face something like this, since you didn't mention it in your post I will comment here. Have you tried to simply request to the third party organization to allow paging?. Get in touch with your third party partner and talk about this.

• If you can, evaluate another source (other third party companies providing a similar service)

Answer 2

Your only option would be to cache the data for some amount of time. If the third party api has a call you can make which is more generic than your dropdown change you could call that before the page loads to get all possible drop down values and store the results for a certain period of time. Then just return the appropriate data each time.

Also, look to see if the third party api has any options for gzipping (compressing) the data before sending it down to you. I have created several API's and have created this type of option for large datasets.

Answer 3

So How I reduce download time or any trick(paging is not supported),Any suggestion?

If the API doesn't support paging, there's not much you could do from a consumer perspective than probably caching the data to avoid the expensive HTTP call.

There's no miracle. There's no client.MaxDownloadTimeMs property that you could set to 5 for example and get your huge data in less than 5 milliseconds.