CODEWITHSUNDEEP
hashredis
user3717756
user3717756

Reputation:

Getting the last 20 hash items in Redis with O(1) time complexity?

Following the Redis suggestion that I must use hashes whenever possible:

Small hashes are encoded in a very small space, so you should try representing your data using hashes every time it is possible. For instance if you have objects representing users in a web application, instead of using different keys for name, surname, email, password, use a single hash with all the required fields.

and that I'll be having millions items, I'm trying to store posts as hashes:

hmset post:1 title "hello"

hmset post:2 title "hi"

hmset post:3 title "g'day"

Now, I know I can retrieve a hash as easy as I created it:

hgetall post:1

The problem is that I want to retrieve more than one, possibly 20 in one row.

I tried the following command which didn't work (without the 3 dots of course):

hgetall post:1 post:2 post:3 post:4...

Giving the following error:

(error) ERR wrong number of arguments for 'hgetall' command

I could do this client side:

hgetall post:1

then

hgetall post:2

and so on.. but that would be a huge waste of network resources and a potentially performance bottleneck as the system should be able to handle at least 5000 users/sec concurrently on average.

Failing what I did, what is the best way to accomplish this? Is there a way to send this commands one time instead of doing it sequentially? Is there a lrange type operation that can be made on hashes?

Upvotes: 2

Views: 972

Answers (2)

user3717756
user3717756

Reputation:

As you pointed out it turns out you can use a Pipelined request to send multiple commands at once.

From the Redis page dedicated to Pipelining:

A Request/Response server can be implemented so that it is able to process new requests even if the client didn't already read the old responses. This way it is possible to send multiple commands to the server without waiting for the replies at all, and finally read the replies in a single step.

This is called pipelining, and is a technique widely in use since many decades. For instance many POP3 protocol implementations already supported this feature, dramatically speeding up the process of downloading new emails from the server.

If I'm not wrong it is the same technique used by the HMSET command, which "Pipelines" multiple HSET commands.

A snippet from the Redis site shows an example with and without Pipelining using Ruby:

require 'rubygems'
require 'redis'

def bench(descr)
    start = Time.now
    yield
    puts "#{descr} #{Time.now-start} seconds"
end

def without_pipelining
    r = Redis.new
    10000.times {
        r.ping
    }
end

def with_pipelining
    r = Redis.new
    r.pipelined {
        10000.times {
            r.ping
        }
    }
end

bench("without pipelining") {
    without_pipelining
}
bench("with pipelining") {
    with_pipelining
}

Go also has a client called Redigo, capable of command Pipelining:

// Initialize Redis (Redigo) client on port 6379 
//  and default address 127.0.0.1/localhost
client, err := redis.Dial("tcp", ":6379")
  if err != nil {
  panic(err)
}
defer client.Close()

// Initialize Pipeline
client.Send("MULTI")

// Send writes the command to the connection's output buffer
client.Send("HGETALL", "post:1") // Where "post:1" contains " title 'hello' content 'hi' "

client.Send("HGETALL", "post:2") // Where "post:1" contains " title 'g'day' content 'g'mornin' "

// Execute the Pipeline
pipe_prox, err := client.Do("EXEC")

if err != nil {
  panic(err)
}

log.Println(pipe_prox)

This is fine as long as you're comfortable showing non-string results.. I'm still trying to find a way to convert the result to strings, as of now I'm getting this:

[[[116 105 116 108 101] [104 105]] [[116 105 116 108 101] [104 101 108 108 111]]]

Doing the following doesn't help either:

result, _ := redis.Strings(pipe_prox, err)

log.Println(pipe_prox)

As this is what I get:

[]

I'll update this answer as soon as I manage to resolve this issue.

Upvotes: 1

Itamar Haber
Itamar Haber

Reputation: 50052

You are in the right direction - do your HGETALLs one after the other for each of the 20 posts. However, to save on network round trips, look into using Redis' pipelining that essentially allows you to group/batch your requests. Alternatively, you can achieve a similar result by implementing you own version of HGETALL that accepts multiple key names with a server-side Lua script.

Upvotes: 1

Related Questions