CODEWITHSUNDEEP
c#databasecachingsynchronization
KhanS
KhanS

Reputation: 1195

Caching huge data into the disk

I have a requirement where huge amount of data needs to be cached on the disk. Whenever there is a change in the database, the data is retreived from the database and cached on the disk. I will be having a background process which keeps checking my cached data with the data base, and updates it as and when required.

I would like to know what would be the best way to organize the cached data on my disk, so that writing and reading from the cache can be faster. An another thread would be used to fetch some new data from the db and cache it on the disk. I also need to take care of synchronization between the two threads.(one will be updating the existing cache data, and the other will be writing newly fetched data into the cache.) Please suggest a strategy for organizing the data on the cache and also synchronization between the threads.

Upvotes: 0

Views: 2085

Answers (3)

csharptest.net
csharptest.net

Reputation: 64218

As an alternative answer, my own B+Tree implementation will neatly address this as a completely managed code (C#) implementation of an IDictionary<TKey, TValue>. It is a single-file key/value store that is thread-safe and optimized for concurrency. It was built from the ground up expressly for this purpose and for providing write-through caches.

Upvotes: 0

Suresh
Suresh

Reputation: 183

Do you care about the consistancy of the data? on power failures?

Memory mapped files along with occational flushes porbably get want you want

Do you need to have an indexed access to data?

You probably need to design something B-tree implementation or B+tree implementation. which gives efficient retrival of the indexed data and better block level locking.

http://code.google.com/p/high-concurrency-btree/

Upvotes: 1

mevdiven
mevdiven

Reputation: 1902

SQL Server has something called XML tables. Those tables are based on physical XML files located in the disk. You can map/link XML data in the disk to a table in SQL Server. For users, it is seamless, in other words they see those tables as a regular tables.

Besides technical/philosophical discussion about caching huge data on the disk, this is just an idea...

Upvotes: 1

Related Questions