Redis failover resulting in unexpected timeouts and errors

Products

• CA API Gateway

Issue/Introduction

Testing failover scenarios with Redis enterprise solution where it involves putting the Redis on maintenance (switching the current master to a slave behind the proxy (single proxy mode)). 

Unexpected events logged on the Gateway:

2025-03-13T15:52:00.170+0000 WARNING 6605  io.lettuce.core.protocol.ConnectionWatchdog: Cannot reconnect to [<redis-Hostname><unresolved>:port]: connection timed out: redis-<hostname>/xxx.xxx.xxx.xxx:port

2025-03-14T18:52:04.440+0000 WARNING 1015  com.l7tech.external.assertions.throughputquota.server.ServerThroughputQuotaAssertion: 4: Unexpected error while updating counter: Redis command timed out; nested exception is io.lettuce.core.RedisCommandTimeoutException: Command timed out after 5 second(s).  Exception caught! 

 

Environment

Gateway 11.1.1

Cause

Redis configured ias single-proxy 

Resolution

Feedback from Redis support, the timeout/delay is expected in single-proxy mode 

Recommendation Redis environment choose multi proxy endpoint.  To setup/configure need to contact Redis support for more information 

Example: <redis-cluster-hostname.net>

Node 1/Secondary = XXX.XXX.XXX.XXX   
Node 2/Primary = YYY.YYY.YYY.YYY
Node 3/Secondary = ZZZ.ZZZ.ZZZ.ZZZ

Also recommend a lower command timeout:

commandTimeout: 500

connectTimeout: 1000

Gateway:  sharedstate_client.yaml

/opt/SecureSpan/Gateway/node/default/etc/bootstrap/providers/sharedstate_client.yaml

redis:

  default:

    testOnStart: false

    type: standalone

    keyPrefixGroupName: layer7

    username: default

    # password: "<password>"

    # encodedPassword: "<encoded_password>"

    commandTimeout: 500

    connectTimeout: 1000

    standalone:

      host: <redis-cluster-hostname.net>

      port: <port>

#    ssl:

#      enabled: true

#      cert: host.crt

#      verifyPeer: false