CODEWITHSUNDEEP
c#asp.net-corecachingredisstackexchange.redis
Eugeniy Maksimov
Eugeniy Maksimov

Reputation: 1704

Does StackExchange Redis Cache recreate connection on Error in the SetAsync method?

We use the Redis cache for our web-api app. We use the Microsoft.Extensions.Caching.StackExchangeRedis. After some works were made on Redis server, our app has failed to write to the cache:

"READONLY You can't write against a read only replica"

If I understand correctly, it is because connection is cached by the redis client (in the Microsoft.Extensions.Caching.StackExchangeRedis code).

SetAsync method calls the OnRedisError method in case of error:

...
        catch (Exception ex)
        {
            OnRedisError(ex, cache);
            throw;
        }
...

In the OnRedisError method the cached connection can be nulled.

private void OnRedisError(Exception exception, IDatabase cache)
{
    if (_options.UseForceReconnect && (exception is RedisConnectionException or SocketException))
    {
...
        // wipe the shared field, but *only* if it is still the cache we were
        // thinking about (once it is null, the next caller will reconnect)
        var tmp = Interlocked.CompareExchange(ref _cache, null, cache);
...
    }
}

And finilly, the UseForceReconnect is

private bool? _useForceReconnect;
    internal bool UseForceReconnect
    {
        get
        {
            return _useForceReconnect ??= GetDefaultValue();
            static bool GetDefaultValue() =>
                AppContext.TryGetSwitch("Microsoft.AspNetCore.Caching.StackExchangeRedis.UseForceReconnect", out var value) && value;
        }
        set => _useForceReconnect = value;
    }

But it seems reconnection in case of error in the SetAsync method doesn't work by default. Also it is written here: https://learn.microsoft.com/en-us/azure/architecture/best-practices/retry-service-specific#retry-usage-guidance-1

The StackExchange Redis client manages its own retries, but only when establishing a connection to the cache when the application first starts. You can configure the connection timeout, the number of retry attempts, and the time between retries to establish this connection, but the retry policy doesn't apply to operations against the cache.

So it seems there is the feature, which is disabled. Can I just enable it or it is a bad idea?

Is there another way to recreate connection in case of error?

PS: the full code of the client: https://github.com/dotnet/aspnetcore/blob/main/src/Caching/StackExchangeRedis/src/RedisCache.cs#L275

Upvotes: 0

Views: 53

Answers (1)

Zhi Lv
Zhi Lv

Reputation: 21636

Yes, you are right that Redis connection is cached.

By default, UseForceReconnect isn't enabled, and the StackExchange Redis client handles retries only during initial connection, not for operations. If there's a failure during a SetAsync, the client won't retry unless we implement some retry logic ourselves.

So, you can try to set the UseForceReconnect property to true and enable the Force Reconnect:

Microsoft.AspNetCore.Caching.StackExchangeRedis.UseForceReconnect = true

That might help the client to reset the connection when an error occurs, which could resolve the READONLY issue after a failover. More detail information, see Using ForceReconnect with StackExchange.Redis

But enabling UseForceReconnect might have some implications. For example, forcing a reconnect every time there's an error could lead to performance issues if it happens too frequently. However, in a failover scenario, it's necessary to reconnect to get the updated primary node.

So, you can also try to use the StackExchange.Redis best practices:

  • Set AbortConnect to false, then let the ConnectionMultiplexer reconnect automatically.
  • Use a single, long-lived ConnectionMultiplexer instance rather than creating a new connection for each request.

Upvotes: 1

Related Questions