CODEWITHSUNDEEP
redisredis-clusterioredis
user906573
user906573

Reputation: 733

Redis (node using ioredis) clusterRetryStrategy exception handling

I have a local setup on a mac running a redis cluster with 6 nodes. I have been trying to handle connection errors and limit the number of retries whenever the cluster is offline (purposefully trying to provoke a fallback to mongo) but it keeps retrying regardless of the cluster / redis options I pass in unless I return "null" in the clusterRetryStrategy. Here is a snapshot of my redis service.

import Redis from 'ioredis';

export class RedisService {
    public readonly redisClient: Redis.Cluster;
    public readonly keyPrefix = 'MyRedisKey';

    public readonly clusterOptions: Redis.ClusterOptions = {
        enableReadyCheck: true,
        retryDelayOnClusterDown: 3000,
        retryDelayOnFailover: 30000,
        retryDelayOnTryAgain: 30000,
        clusterRetryStrategy: (times: number) => {
            // notes: this is the only way I have figured how to throw errors without retying for ever when connection to the cluster fails. 
            // tried to use maxRetriesPerRequest=1, different variations of retryStrategy
            return null;
        },
        slotsRefreshInterval: 10000,
        redisOptions: {
            keyPrefix: this.keyPrefix,
            autoResubscribe: true,
            autoResendUnfulfilledCommands: false,
            password: process.env?.REDIS_PASSWORD,
            enableOfflineQueue: false,
            maxRetriesPerRequest: 1
        }
    };

    constructor() {
        const nodes: any = ['127.0.0.1:30001','127.0.0.1:30002','127.0.0.1:30003','127.0.0.1:30004','127.0.0.1:30005','127.0.0.1:30006'];
        this.redisClient = new Redis.Cluster(nodes, this.clusterOptions);
    }

    public hget = async (field: string): Promise<any> => {
        const response = await this.redisClient.hget(this.keyPrefix, field);
        return response;
    }
}

If my redis cluster is stopped and I make a call to this service like:

public readonly redisService: RedisService = new RedisService();
try {
  const item = await this.redisService.hget('test');
} catch(e){
  console.error(e);
}

I endlessly get the error "[ioredis] Unhandled error event: ClusterAllFailedError: Failed to refresh slots cache." and it never falls into the catch block.

By the way, I tried the solution listed here but it did not work. Redis (ioredis) - Unable to catch connection error in order to handle them gracefully.

Below are the versions of ioredis npm packages I am using.

"ioredis": "4.19.4"
"@types/ioredis": "4.17.10"

Thank you for your help.

Upvotes: 4

Views: 2315

Answers (1)

M07
M07

Reputation: 1131

Refresh slots cache is done automatically by IoRedis. Setting them to 10000ms is bad because it means than any new shards added into your cluster will not be visible by your NodeJs sever for 10s (that could create some nasty errors in run time)

In order, to catch those errors you can listen on error events.

this.redisClient.on('error', async (error: any) => {
    logger.error(`[EventHandler] error`, { errorMessage: error.message });
});

About the clusterRetryStrategy, it is true that it a bit odd. From what I discover, IoRedis calls this clusterRetryStrategy function as follow

  1. a couple of times without an error in the second param
  2. a lot of time with actual error is passed in param getaddrinfo ENOTFOUND
  3. for ever without the error when connection is back to normal

In order to stop the infinit loop when connection is back, I use the following function:

import Redis from 'ioredis';

export class RedisService {
    public readonly redisClient: Redis.Cluster;
    public readonly keyPrefix = 'MyRedisKey';
    private retryStrategyErrorDetected: boolean = false;

    public readonly clusterOptions: Redis.ClusterOptions = {
        clusterRetryStrategy: (times: number, reason?: Error) => {
            tLogger.warn(`[EventHandler] clusterRetryStrategy`, { count: times, name: reason?.message});    
            if (this.retryStrategyErrorDetected && !reason) {
              this.retryStrategyErrorDetected = false;
              return null;
            }
            if (reason) {
              this.retryStrategyErrorDetected = true;
            }
            return Math.min(100 + times * 2, 2000);
          };
        },
    };

    constructor() {
        const nodes: any = ['127.0.0.1:30001','127.0.0.1:30002','127.0.0.1:30003','127.0.0.1:30004','127.0.0.1:30005','127.0.0.1:30006'];
        this.redisClient = new Redis.Cluster(nodes, this.clusterOptions);
        this.redisClient.on('error', async (error: any) => {
            logger.error(`[EventHandler] error`, { errorMessage: error.message });
        });
    }
}

Hope it will help someone.

Upvotes: 1

Related Questions