CODEWITHSUNDEEP
goredisredigo
Calvin Brizzi
Calvin Brizzi

Reputation: 25

Redigo: Fail fast when redis server is down

I'm struggling with getting go to fail fast when the redis server I'm connected to goes down, want to have a robust solution.

I'm using redigo and I'm setting up a connection pool like so:

// This has other stuff in it in the code, use it as a 
// central repository for things we want in memory
type State struct{
    redisPool *redis.Pool
}

func (state *State) GetRedisConn() redis.Conn {
    return state.redisPool.Get()
}

func main() {
    state.redisPool = &redis.Pool{
        MaxIdle:     200,
        MaxActive:   9000,
        IdleTimeout: time.Minute,
        Dial: func() (redis.Conn, error) {
            return redis.Dial("tcp", *redisAddress,
                redis.DialConnectTimeout(1*time.Second),
                redis.DialReadTimeout(100*time.Millisecond),
                redis.DialWriteTimeout(100*time.Millisecond),
            )
        },
    }
}

And requesting new connections and using them like so:

t0 := time.Now()

conn := state.GetRedisConn()
if conn != nil && conn.Err() == nil {
    defer conn.Close()
    // Do stuff
else {
    log.Printf("no redis probably")
}

log.Println(time.Now().Sub(t0).Seconds())

While redis is up, this works great, things happen in milliseconds. The moment I take redis down my 75th percentile goes up to 7+ seconds, with my 99th percentile going up to 10s (I can see this on prometheus)

What am I doing wrong? Why does this not timeout faster? I was under the impression that redis.DialConnectTimeout(1*time.Second) would cap the issue at 1 second, but it doesn't seem to be the case.

EDIT: It turns out this was due to a mistake I was making in Prometheus, setting the buckets too big, so while redis was timing out fine after a second, my buckets had been set up with a 1s bucket and a 10s bucket, so my requests (which were just over 1s) ended up in the 10s bucket, skewing the results. I'm sure this discussion will be useful to someone at some point though.

Upvotes: 1

Views: 1945

Answers (1)

Thundercat
Thundercat

Reputation: 120941

Rate limit dial attempts after a failure:

func main() {
    var (
       nextDial time.Time
       mu sync.Mutex
    )
    state.redisPool = &redis.Pool{
        MaxIdle:     200,
        MaxActive:   9000,
        IdleTimeout: time.Minute,
        Dial: func() (redis.Conn, error) {
            mu.Lock()   // Dial can be called concurrently
            defer mu.Unlock()
            if time.Now().Before(nextDial) {
               return nil, errors.New("waiting for dial")
            }
            c, err := redis.Dial("tcp", *redisAddress,
                redis.DialConnectTimeout(1*time.Second),
                redis.DialReadTimeout(100*time.Millisecond),
                redis.DialWriteTimeout(100*time.Millisecond),
            )
            if err == nil {
               nextDial = time.Time{}
            } else {
               nextDial = time.Now().Add(time.Second) // don't attempt dial for one second
            }
            return c, err
        },
    }
}

Upvotes: 2

Related Questions