CODEWITHSUNDEEP
riakleveldb
Tremo
Tremo

Reputation: 599

Is there an efficient way to process data expiration / massive deletion (to free space) with Riak on leveldb?

On Riak :

  1. Is there a way to process data expiration or to dump old data to free some space?
  2. Is it efficient ?

Edit: Thanks to Joe to provide the answer and its workaround (answer down). Data expiration should be thought from the very beginning as it requires an additional index with a map-reduce algorithme.

Upvotes: 0

Views: 798

Answers (2)

Daniel Walter
Daniel Walter

Reputation: 825

I have written a little c++ tool that uses the leveldb internal function CompactRange to perform this task. Here you can read the article about this.
With this we are able to delete an entire bucket (key by key) and wipe all tombstones. 50Gb of 75Gb are freed!
Unfortunately, this only works if leveldb is used as backend.

Upvotes: 1

Joe
Joe

Reputation: 28376

Short answer: No, there is no publisher-provided expiry.

Longer answer: Include the write time, in a integer representation like Unix epoch, in a secondary index entry on each value that you want to be subject to expiry. The run a periodic job during off-peak times to do a ranged 2I query to get any entries from 0 to (now - TTL). This could be used as an input to a map/reduce job to do the actual deletes.

As to recovering disk space, leveldb is very slow about that. When a value is written to leveldb it starts in level 0, then as each level fills, compaction moves values to the next level, so your least recently written data resides on disk in the lowest levels. When you delete a value, a tombstone is written to level 0, which masks the previous value in the lower level, and as normal compaction occurs the tombstone is moved down as any other value would be. The disk space consumed by the old value is not reclaimed until the tombstone reaches the same level.

Upvotes: 3

Related Questions