CODEWITHSUNDEEP
redisjedis
Ali Salehi
Ali Salehi

Reputation: 6999

Jedis Storage Confusion, Is this issue from Redis or Jedis?

The following java code inserts a million integer pairs into redis. 

 public class JedisInsertion { 
        public static byte[] fromInt(int v) { 
                return ByteBuffer.allocate(4).putInt(v).array(); 
        } 
        public static void main(String args[]) { 
                Jedis j = new Jedis("localhost"); 
                for (int i = 0;i<1000*1000;i++){ 
                   j.set(fromInt(i),fromInt(i)); 
                } 
        } 
}

here is redis info output 

... 
used_memory:89319664 
arch_bits:64 
...

89319664 implies ~ 89 bytes per key value pair.

I was expecting something around 8 MB instead (4 bytes for key + 4 bytes value).

I also compiled redis in 32 bit mode (still running the test on a 64 bit machine).

The results for 32-bit version of redis:

used_memory: 68831664 => 68 bytes per key value pair.

Both results are several times higher than what I was expecting.

When I look at redis using CLI, I see a typical key would look like this: "\x00\x00\xc2\xff"

I appreciate your feedback

PS - I am using Redis 2.2.14 and Jedis 2.0 on a 64-bit machine

PSS - I also tried to store all the values in one hash as suggested by one of the comments, here is the code:

for (int i = 0;i<1000*1000;i++){
    j.hset("my-hash".getBytes(),fromInt(i),fromInt(i));
}

And here is the results:

used_memory_rss:84676608 (for 32bit build)
used_memory:105319712 (for 64bit build)

The results are even worse when I use a single hash.

Upvotes: 0

Views: 727

Answers (2)

Carl Zulauf
Carl Zulauf

Reputation: 39558

The fact that your key is a non-readable binary value is a decision made by Jedis, since Redis can use pretty much anything as a key.

Redis keys have expirations and other meta-data that takes up additional space. If you are concerned about space, consider storing all your values within a single Redis hash. There is significantly less overhead for key/value pairs within a hash than for standard key/value pairs.

http://redis.io/topics/memory-optimization

Upvotes: 0

antirez
antirez

Reputation: 18504

68 byes per key/value type are correct. Redis does not internally store things as a plain text file, otherwise it would not be able to lookup it in a fast way, to have different types of objects, and so forth. There is an associated overhead. There are more informations about that in the documentation at Redis.io.

Upvotes: 1

Related Questions