CODEWITHSUNDEEP
delphitclientdataset
Svip
Svip

Reputation: 3138

TClientDataSet and limit by memory

We have a system that creates reports out of our data. And we can deal with a lot of data. The idea of over 150,000 rows is not out of the question.

Unfortunately, our experience with TClientDataSet is its limitations, because it often results in an 'insufficient memory for this operation' error, when the data gets too big.

So the question is this: Does there exist a generally available implementation of TDataSet that can handle a large amount of data (such as streaming directly to a file and not keeping the entire dataset in memory)?

I am willing to implement such a class myself. But as far as I understand TClientDataSet, it needs to be able to contain the data itself before it can save it to a file/stream. In addition, loading the data again should also be possible as a stream rather than loading in an entire TClientDataSet object, because then we wouldn't have solved the issue.

Upvotes: 2

Views: 4365

Answers (4)

altink
altink

Reputation: 367

Wanted to underline that the capacities of the TClientDataset could be something bigger.

Test on the limits of TClientDataset - appending xxx,xxx records, putting the single record in whole (repeated) to create an idea on the size.

// Begin Load Record to TCLientDataset for Reverse (on Reverse) Processing

dxMemData1.Append;

dxMemData1['NT_Rec_No'] := 1000;

dxMemData1['NT_User'] := 'DEV\Administrator';

dxMemData1['NT_Type'] := 'Information';

dxMemData1['Ora_Timestamp'] := '20170706033859.000';

dxMemData1['Ora_Host'] := 'DEV0001';

dxMemData1['Ora_SID'] := 'Oracle.orcl';

dxMemData1['Ora_Event_Id'] := '34';

dxMemData1['NT_Message'] := Memo1.Text;

dxMemData1.Post;

// End Load Record to TCLientDataset for Reverse (on Reverse) Processing

String on the memo1 is a of 100 characters - ansi

did several tests and managed to keep going to something from 600,000 to 900,000 records without crashing.

Difference can be made by making the text on the memo bigger - this did decrease the max number before crash - which means it is not a matter of exact max. record number - but of size consumed - my guess.

Tested the same with TdxMemData (devexpress), this time I could reach almost the double of records

Upvotes: 0

Pascal D
Pascal D

Reputation: 258

KBMMemTable is a nice alternative to TClientDataset

http://www.components4programmers.com/products/kbmmemtable/

We are using it for years and it is very useful and fast.

Upvotes: 0

afrazier
afrazier

Reputation: 4902

Is there really any need to cache all the data on the client before reporting? If not, maybe rethink how you're querying and processing data to generate these reports and see if there's a way that involves less client-side data (which comes with a bonus of less data transmitted over the network).

If you've been down that road before and you really do need all this data client side, then you could look at custom data structures. A TList<T> of records (even if you need to build your own indexes) takes a lot less memory than a TClientDataSet does.

Upvotes: 1

iMan Biglari
iMan Biglari

Reputation: 4786

You can use either FireBird or Interbase in embedded mode.

Upvotes: 4

Related Questions