CODEWITHSUNDEEP
pythondatabasedjangopostgresqlweb
StarLegend
StarLegend

Reputation: 35

Handle and display large data set in web browser

I am still a noob in web app development and sorry if this question might seem obvious for you guys.

Currently I am developing a web application for my University using Python and Django. And one feature of my web app is to retrieve a large set of data in a table in the database(postgreSQL), and displaying these data in a tabular form on webpage. Each column of the table need to have the sorting and filtering feature. The data set goes up to roughly 2 millions of rows.

So I wonder if something like jpxGrid could help me to achieve such goal or it would be too slow to handle/sort/display/render such a large data set on web page. I plan to retrieve all the data inside the table once (only initiate one database query call) and pass it into jpxGrid, however, my colleague suggests that each sort and filter should initiate a separate query call to the database to achieve better performance(database order by is very fast). I tried to use another open source jquery library that handles the form and enables sorting, filtering and paging(non professional outdated one) at the beginning, which starts to lag after 5k data rows and becomes impossible to use after 20k rows.

My question is if something like jpxGrid is a good solution to my problem or I should build my own system that letting the database to handle the sorting and filtering(probably need to add the paging feature too). Thank you very much for helping.

Upvotes: 1

Views: 1663

Answers (1)

Sean Azlin
Sean Azlin

Reputation: 916

Are you allowed to use paging in your output? If so, then i'd start by setting a page size of 100 (for example) and then use LIMIT 100 in my various SQL queries. Essentially, each time the user clicks next or prev on the web page a new query would be executed based on the current filtering or sorting options with the LIMIT. The SQL should be pretty easy to figure out.

Upvotes: 3

Related Questions