CODEWITHSUNDEEP
redis
Duru Can Celasun
Duru Can Celasun

Reputation: 1681

Finding non-expiring keys in Redis

In my setup, the info command shows me the following:

[keys] => 1128
[expires] => 1125

I'd like to find those 3 keys without an expiration date. I've already checked the docs to no avail. Any ideas?

Upvotes: 64

Views: 49521

Answers (8)

maf-soft
maf-soft

Reputation: 2552

I wanted a fast Lua script, but with the recommended SCAN command and a nice output. It contains some example conditions, so you can easily adapt it to your needs.

local maxcount = 10000
local result, cursor, count, total_memory = {}, '0', 0, 0

repeat
    local scan_result = redis.call('SCAN', cursor, 'COUNT', 100)
    cursor = scan_result[1]

    for _, key in ipairs(scan_result[2]) do
        if string.match(key, '^prefix') then
            local ttl = redis.call('TTL', key) / 3600
            local idle_time = redis.call('OBJECT', 'IDLETIME', key) / 3600

            if (ttl < 0 or ttl > 24) and idle_time > 24 then
                local memory_usage = redis.call('MEMORY', 'USAGE', key)
                --[[ comment the following line to only get the summary ]]
                table.insert(result, string.format('%.1fh %.1fh %.1fkb %s', ttl, idle_time, memory_usage / 1024, key))
                total_memory = total_memory + memory_usage
                count = count + 1
                --[[ redis.call('DEL', key) ]]
            end

            if count >= maxcount then
                break
            end
        end
    end
until cursor == '0' or count >= maxcount

table.insert(result, string.format('Count: %d, Size: %.1fmb', count, total_memory / (1024 * 1024)))
return result

You can run it like that:

$ redis-cli --eval filename.lua

I only had RedisInsight installed on Windows, it has a built-in CLI, but I had to convert it to a one-liner (just replace line breaks by spaces) and run EVAL "..." 0.

It took (without the delete) about 20s to search 4M keys and return 10K of them in my case.

Upvotes: 0

azmirfakkri
azmirfakkri

Reputation: 601

I was trying to accomplish the same at work, where I need to find keys without TTL set and then delete them.

This script is written in Java, I ran it using my IDE.

  1. Check the port number is correct (it needs to be the same port number opened when access to the cluster is established)
  2. Check that the Redis key name that you want to match on is correct
import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisPool;
import redis.clients.jedis.JedisPoolConfig;
import redis.clients.jedis.params.ScanParams;

class RemoveRedisKeysWithoutTtl {

  public static void main(String[] args) {
    JedisPool jedisPool = new JedisPool(new JedisPoolConfig(), "localhost", 6379);

    try (Jedis jedis = jedisPool.getResource()) {
      // call the method to remove keys with no TTL set
      removeKeysWithNoTTL(jedis);

      System.out.println("Keys with no TTL removed successfully.");
    }

    // close the connection
    jedisPool.close();
  }

  public static void removeKeysWithNoTTL(Jedis jedis) {
    // update the pattern to match on your key
    ScanParams scanParams = new ScanParams().match("*");

    // start at the beginning of the cursor, so value is 0
    String cursor = ScanParams.SCAN_POINTER_START;

    do {
      var scanResult = jedis.scan(cursor, scanParams);
      var keys = scanResult.getResult();

      // iterate through keys
      for (String key : keys) {
        // check if TTL is -1 (no expiration) or -2 (key does not exist)
        if (jedis.ttl(key) == -1 || jedis.ttl(key) == -2) {
          // Delete the key
          System.out.println("Deleting key " + key);
          jedis.del(key);
        }
      }

      // get the next cursor
      cursor = scanResult.getCursor();
    } while (!cursor.equals(ScanParams.SCAN_POINTER_START));
  }
}

Upvotes: 0

aron23
aron23

Reputation: 323

I needed to extract non-expiring keys from bigger (40GB) dataset, so using keys command was not suitable for me. So in case someone is looking for offline/non-blocking solution, you can use https://github.com/sripathikrishnan/redis-rdb-tools for extraction of non-expiring keys from redis rdb dump.

You can just install libraries via pip:

pip install rdbtools python-lzf

Then create simple parser which extracts keys and values which has expiration set to None:

from rdbtools import RdbParser, RdbCallback
from rdbtools.encodehelpers import bytes_to_unicode

class ParserCallback(RdbCallback):

    def __init__(self):
        super(ParserCallback, self).__init__(string_escape=None)

    def encode_key(self, key):
        return bytes_to_unicode(key, self._escape, skip_printable=True)

    def encode_value(self, val):
        return bytes_to_unicode(val, self._escape)

    def set(self, key, value, expiry, info):
        if expiry is None:
            print('%s = %s' % (self.encode_key(key), self.encode_value(value)))


callback = ParserCallback()
parser = RdbParser(callback)
parser.parse('/path/to/dump.rdb')

Upvotes: 3

Bluehorn
Bluehorn

Reputation: 3091

To me the accepted answer appears unusable for a medium-sized dataset, as it will run a redis-cli command for each and every key.

Instead I used this lua script to filter the keys inside the redis server:

local show_persistent = ARGV[1] ~= "expiring"

local keys = {}
for i, name in ipairs(redis.call("keys", "*")) do
    local persistent = redis.call("pttl", name) < 0
    if persistent == show_persistent then
        table.insert(keys, name)
    end
end

return keys

This can be called as

$ redis-cli --eval show-persistent-keys.lua

to get all keys without an expiration time. It also can be called as

$ redis-cli --eval show-persistent-keys.lua , expiring

to find the opposite key set of all keys with an expiration time set.

On the downside this may block for too long (appears fine for 1 M keys). I'd use scan instead but I happen to have to run this against a legacy Redis at version 2.6, which does not have scan available.

Upvotes: 5

teastburn
teastburn

Reputation: 3488

@Waynn Lue's answer runs but uses the Redis KEYS command which Redis warns about:

Warning: consider KEYS as a command that should only be used in production environments with extreme care. It may ruin performance when it is executed against large databases.

Redis documentation recommends using SCAN.

redis-cli --scan | while read LINE ; do TTL=`redis-cli ttl "$LINE"`; if [ $TTL -eq  -1 ]; then echo "$LINE"; fi; done;

If you want to scan for a specific key pattern, use:

 redis-cli --scan --pattern "something*"

Upvotes: 67

destator
destator

Reputation: 171

#!/usr/bin/env python

import argparse
import redis

p = argparse.ArgumentParser()
p.add_argument("-i", '--host', type=str, default="127.0.0.1", help="redis host", required=False)
p.add_argument("-p", '--port', type=int, default=6379, help="redis port", required=False)
p.add_argument("-n", '--db', type=int, default=0, help="redis database", required=False)

args = p.parse_args()

r = redis.Redis(host=args.host, port=args.port, db=args.db)

try:
    keys = r.keys()

    for key in keys:
        if r.ttl(key) < 0:
            print(key)
except KeyboardInterrupt:
    pass

Upvotes: -1

Waynn Lue
Waynn Lue

Reputation: 11375

Modified from a site that I can't find now.

redis-cli keys  "*" | while read LINE ; do TTL=`redis-cli ttl "$LINE"`; if [ $TTL -eq  -1 ]; then echo "$LINE"; fi; done;

edit: Note, this is a blocking call.

Upvotes: 71

zephyr
zephyr

Reputation: 1900

In case somebody is getting bad arguments or wrong number of arguments error, put double quotes around $LINE.

So,it would be

redis-cli keys  "*" | while read LINE ; do TTL=`redis-cli ttl "$LINE"`; if [ $TTL -eq  -1 ]; then echo "$LINE"; fi; done;

This happens when there are spaces in the key.

Upvotes: 4

Related Questions