CODEWITHSUNDEEP
goredisgo-rediskeydb
Sergey Klimov
Sergey Klimov

Reputation: 23

Application not changing master on Sentinel + KeyDB pair dropdown

Description

I have 3 pairs of Sentinel + KeyDB working on separated machines. Sentinels connected between each other so they can observe dropdowns. Also, I have an application, which uses go-redis to connect to Sentinels. Connecton code:

client := redis.NewFailoverClient(&redis.FailoverOptions{
        MasterName:       "master",
        SentinelAddrs:    []string{"addr1:26379","addr2:26379","addr3:26379"},
        SentinelPassword: "pass",
        Password:         "pass",
    })

In case when KeyDB master gets down, appication understands it and make a connection to new master logging this message:

sentinel: new master="master" addr="addr2:6379"

In case when observed Sentinel gets down, appication understands it and make a connection to new Sentinel logging this message:

redis: discarding bad PubSub connection: EOF

Problem

I've got a case, when a pair Sentinel + KeyDB (addr1:26379), observed by an application, drops down together. At the moment, application logs this:

redis: discarding bad PubSub connection: EOF

But then, when I try to make a request via client, I get an error like this:

dial tcp addr1:26379: i/o timeout

For me it seems like by some reason Failover Client not switching Sentinels (other 2 pairs of Sentinel + KeyDB continue working with new master), when an observed pair falls down.

Question

What did I miss, so the propably common case makes this behavior? Is it my fault or I should open an issue in go-redis or key-db?

I've tried client different client options like timeouts, but nothing force client to change Sentinel address.

I expect client to understand that Sentinel is down and switch to another instance and get new master.

Upvotes: 1

Views: 85

Answers (0)

Related Questions