Gateway application unable to failover to secondary Gateway database

Products

• STARTER PACK-7
• CA Rapid App Security
• CA API Gateway

Issue/Introduction

The following error may be seen in the Layer 7 Gateway log files when a Gateway does not appear to be failing over to the secondary Gateway database:

org.hibernate.util.JDBCExceptionReporter: SQL Error: 0, SQLState: null
org.hibernate.util.JDBCExceptionReporter: Connections could not be acquired from the underlying database!

Environment

Release:
Component: APIGTW

Cause

Typically, this occurs because a Gateway database node's local MySQL server was terminated after the Gateway was initialized. For example: Gateway 1 is started successfully and is able to process traffic. Gateway 1's MySQL server is then terminated forcefully by an operator. Gateway 1 will then begin recording the error messages logged above. In a normal operating environment, a single Gateway database node running without failover or replication will cease processing traffic. In order to maintain continuity of service, the Gateway is capable of failing over to a specific secondary Gateway database--typically present on an additional Gateway instance. This is considered clustering.

Resolution

In order to properly cluster two Gateway database nodes into one cohesive group, they must be configured to do so. If you have two Gateway database nodes that should be clustered and are not appearing to failover properly then verify that each Gateway is configured with a Database Failover Connection. This can be done as follows:

1. Log into the Layer 7 Gateway as the ssgconfig user.
2. Select Option #2: Display Layer 7 Gateway configuration menu.
3. Select Option #6: Display the current Layer 7 Gateway configuration.
4. Fields under Configure Database Failover Connection should be populated.

If Database Failover Connection is configured per the above then each Gateway will need to be configured for clustering and failover:

1. Log into the Gateway as the ssgconfig user. 
2. Select Option #2: Display Layer 7 Gateway configuration menu.
3. Select Option #3: Configure the Layer 7 Gateway.
4. Select Option #2: Configure Database Failover Connection
5. Proceed through the prompts to configure a failover connection on each Gateway.

Once this is complete, the Gateway service should be restarted on each appliance.

1. Select Option #7: Manage Layer 7 Gateway status.
2. Select Option #2: Restart the Layer 7 Gateway.

Once the Gateway service has been restarted on each appliance, failover should function properly.

Additional Information

When a failure of the primary database is detected by the API Gateway, new connections to the secondary database from the API Gateway are established.
After a recovery of the primary database, these connections are gradually switched to the primary database along with the API Gateway's demand for new connections.
("Failback" isn't done at once.)