CODEWITHSUNDEEP
cachingredislru
bpeikes
bpeikes

Reputation: 3715

Can Redis use disk as part of a LRU cache?

We have the need for a distributed LRU cache, but one which can use both memory and disk. We have a large dataset, which is stored on disk permenantly. From that dataset, we create other calculated datasets, but only when clients need them.

Since these secondary datasets are derived from data which is persistent, we never need to permanently save this derived data.

I thought that Redis would have the ability to use disk as a secondary LRU cache, but have not been able to find any documentation that points to that. It seems like Redis only uses the disk to persist the entire cache. I envisioned that we'd be able to scale out horizontally with a bunch of Redis instances.

If Redis can not do this, is there another system that does?

Upvotes: 1

Views: 1867

Answers (2)

Matthew O'Riordan
Matthew O'Riordan

Reputation: 8211

The way to solve this problem without changing how your clients work, is in fact not to use Redis, but instead to use a Redis compatible database like Ardb which in turn can be configured to use LevelDB under the hood which supports LRU type on-disk caches.

Upvotes: 1

cruftex
cruftex

Reputation: 5723

If the data does not fit into memory, the OS can swap it out to the disk. This is called virtual memory. Here you find an explanation: http://redis.io/topics/virtual-memory

Remark: You want to retrieve some data, do stuff on it and you have some intermediate results. Please check whether you may want to distribute your processing, not only the data. Take a look at Apache Hadoop and especially Apache Spark.

Upvotes: 1

Related Questions