Gateway is not starting after RDS DB failover due to unable to connect to database

Products

• CA API Gateway
• API SECURITY
• CA API Gateway Precision API Monitoring Module for API Gateway (Layer 7)
• CA API Gateway Enterprise Service Manager (Layer 7)
• STARTER PACK-7
• CA Microgateway

Issue/Introduction

API  Gateway is not starting  after failover. test of the RDS database cluster

The ssg log shows that the gateway is unable to connect to the database server but other tools like mysql can connect .

2020-11-20T08:27:48.132+0000 INFO    1 com.l7tech.server.boot.GatewayBoot: Enabled component: [com/l7tech/server/resources/uddiRuntimeContext.xml, com/l7tech/server/resources/uddiAdminContext.xml, com/l7tech/server/resources/databaseReplicationMonitorRuntimeContext.xml, com/l7tech/server/resources/databaseReplicationMonitorAdminContext.xml, com/l7tech/server/resources/processControllerRuntimeContext.xml, com/l7tech/server/resources/processControllerAdminContext.xml]
2020-11-20T08:28:18.125+0000 SEVERE  12 com.l7tech.server.boot.GatewayBoot: WARNING: No database connections open after 30 seconds; possible DB connection failure?
2020-11-20T08:32:53.947+0000 WARNING 1 org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy: Could not retrieve default auto-commit and transaction isolation settings
java.sql.SQLException: Connections could not be acquired from the underlying database!

And

2020-11-20T14:58:18.343+0000 SEVERE 12 com.l7tech.server.boot.GatewayBoot: WARNING: No database connections open after 30 seconds; possible DB connection failure?

on RDS audit log 

1605884662505562,xxxxxxxxxxxxxx-cluster-new-cluster,,xxxxxxx ,105766,0,FAILED_CONNECT,,,1043 (edited)

the 1043 mysal error indicates a handshake or MySQL client version mismatch .

Environment

Release : 9.4

Component : API GATEWAY

Cause

The tcpdump shows Gateway tries to connect to RDS mysql using TLS1.1 by sending client hello

There is no response from rds aws database on this request and the rds is logging an error 1043..

After checking the database setting tls_version we found this is set to TLS1.2 only .on these rds nodes. These settings became active due to the reboot done during the failover.

For an Aurora MySQL 5.7 DB cluster, you can use the tls_version DB cluster parameter to indicate the permitted protocol versions

Resolution

There are two solutions either update the allowed tls version in rds to allow Tls1.1 you need to reboot the rds dbs to make the setting active 

Or  

updated the gateway node properties file in  /opt/SecureSpan/Gateway/node/default/etc/conf/node.properties

add the line 

l7.mysql.url.parameters.extra=&useSSL=true&verifyServerCertificate=false&enabledTLSProtocols=TLSv1.2

To enforce the mysql jdbc driver is using  TLS 1.2 to connect to MySQL by default.