After DB failover to secondary we are noticing performance issues with more latency and lower TPS

Products

• CA API Gateway

Issue/Introduction

We had a storage issue and this caused the primary database to become unavailable and switch over to the secondary database.

And we are currently having performance issues with more latency and lower TPS.

 

Environment

Release : 10.1

Resolution

There is a slight performance concern when you are using the secondary database.

As the jdbc connection is frequently being tested to primary, as it will want to try and failback as the primary connection. 

So as JDBC connections timeout they will retry the primary then goto secondary db.

And depending on the location of the secondary if its in a different data center it can even add additional overhead.

But the connection always wants to go to primary before secondary i.e. 1,2 in jdbc so this in itself may add some performance latency. If the load is heavier it is sometimes a noticeable performance concern and effects TPS.