CODEWITHSUNDEEP
c#datatable
BuZz
BuZz

Reputation: 17485

Fill a C# DataTable "on disk" rather than "in memory" to avoid Memory overflow

I need to fill two DataTable from two different databases for some comparisons and read-only operations. That takes me to "select *" into these two DataTable.

However, the table I need to retrieve contains about 3 million rows in both databases. I get an OutOfMemoryExceptionwhile filling.

Is there a possibility to have these DataTable on disk rather than in memory? Do you see any other solution to compare/perform read-only on such two huge sets? The things that I need to do make it very complicated to proceed by batches of subsets of rows.

Upvotes: 1

Views: 484

Answers (2)

Mare Infinitus
Mare Infinitus

Reputation: 8192

What about sorting both of them by key.

i.e.

set1    A  A  A  B  C            E
                 |            
set2             B       D

then you can go on set1, change to set2, go back to set1... makes iterating much easier.

Upvotes: 1

Dr. Wily's Apprentice
Dr. Wily's Apprentice

Reputation: 10280

My guess is that the ideal/efficient way of performing this would be to do it within the database. Unfortunately, your two tables are in two separate databases, which may cause complications.

Are these tables in two different databases on the same server (as in SQL Server instance), or are they in two separate servers? If they are within the same server/instance, then you could probably perform your comparison logic on the database server.

Otherwise, if these tables are on separate servers, then is there any possibility of setting up some kind of replication between databases so that you can get both data sets into a single database where you can perform your comparisons?

Upvotes: 1

Related Questions