Azure Production DR Failover - Gateway start up showing abnormal startup times

Products

• CA API Gateway

Issue/Introduction

We have a East and West region production container gateway setup.

Our primary, East, takes ~1-2 minutes to start as expected, though our DR instance in West takes 10-20 minutes.

We have azure replication configured, and the startup for DR get stuck on "Version checking..." at start up.

please let me know if you see something unusual, and what next steps would be.

Environment

Release : 10.0

Cause

We have explained this is because the latency produced on that condition when because the EntityVersionChecker periodically checks to see whether entities have been changed in the Gateway’s configuration database. 
- The slowness Condition is presented only when the database is in West and Gateway on East or vice versa (long geographical distances scenario)
- On startup the gateway is populating policyCache and runtime details, and with a latent DB connection (Gw East -> West DB , Gw West -> East DB ) 
    This is because we can observed some traffic is blocked/unprocessed for a long duration.

- the gateway log files , the West to to East Log shows the same behavior than East to West Region , Version checking took 15172ms

 

Resolution

- On startup the gateway is populating policyCache and runtime details, and with a latent DB connection (Gw East -> West DB ) , that traffic is blocked/unprocessed for a long duration.

- This is confirmed when you used East-East (2-4mins) or West-West (4min) config with a lower response time on the same startup

- Customer used  East-East (2-4mins) or West-West (4min) on cluster Gateway configuration to improve response times.