Handle huge amount of data in Qt

Question

I need to read some text files that contain a huge amount of data, say 4 files each of about 500MB.

Each file contains several lines and each line has about this format:

id timestamp field1 field2 field3 field4

My strategy so far was to parse each file and for every line creating a QTreeWidgetItem with a suitable number of fields to store that line (this because during the program I want to show some of these data in a QTreeWidget) and appending all these items to a QList.

This QList is stored for all the execution of the program, in this way data are always available and I don't need to parse the files anymore.

I need all the data available because at each moment I need to access to data relative to a particular timestamp interval.

However this strategy seems too expansive in terms of resources, because I saw that the program consumes several GBs of memory and it eventually crashes.

How can I approach in a better way the handling of such data?

Answer 1

What you want is called 'lazy loading'. There is an Example in the Qt documentation which shows you, how to use QAbstractItemModel, canFetchMore() and fetchMore().