Many jobs are failing in Prod today due to JDBC connection failure

Products

• CA Test Data Manager (Data Finder / Grid Tools)

Issue/Introduction

Today, we have seen multiple jobs fail due to JDBC connection issues.
Please tell us how to overcome these at the earliest.
When the connection errors occurs, it happens to multiple jobs running and all of them fail.

 

Environment

CA Test Data Manager
TDM Portal

Resolution

Based on the following lines in the TDMWeb.log


Line 178731: Wrapped by: org.springframework.dao.InvalidDataAccessResourceUsageException: Unable to acquire JDBC Connection; SQL [n/a]; nested exception is org.hibernate.exception.SQLGrammarException: Unable to acquire JDBC Connection

Line 178731: Wrapped by: org.springframework.dao.InvalidDataAccessResourceUsageException: Unable to acquire JDBC Connection; SQL [n/a]; nested exception is org.hibernate.exception.SQLGrammarException: Unable to acquire JDBC Connection

Line 179045: 2019-09-17 17:17:31.977 UTC [ERROR] [http-nio-443-exec-1780 ] --- [U:][M:][P:] c.c.t.c.e.h.GlobalExceptionConvertor: HTTP Error - INTERNAL SERVER ERROR: Exception: Unable to acquire JDBC Connection; SQL [n/a]; nested exception is org.hibernate.exception.SQLGrammarException: Unable to acquire JDBC Connection

Line 179045: 2019-09-17 17:17:31.977 UTC [ERROR] [http-nio-443-exec-1780 ] --- [U:][M:][P:] c.c.t.c.e.h.GlobalExceptionConvertor: HTTP Error - INTERNAL SERVER ERROR: Exception: Unable to acquire JDBC Connection; SQL [n/a]; nested exception is org.hibernate.exception.SQLGrammarException: Unable to acquire JDBC Connection

Line 179046: com.microsoft.sqlserver.jdbc.SQLServerException: Unable to access availability database 'GTREP_PD_Shadow' because the database replica is not in the PRIMARY or SECONDARY role. Connections to an availability database is permitted only when the database replica is in the PRIMARY or SECONDARY role. Try the operation again later. ClientConnectionId:a226bff6-c770-46cf-860f-70cd177de58b

2019-09-17 17:17:31.977 UTC [ERROR] [http-nio-443-exec-1773 ] --- [U:C211308][M:POST][P:/api/ca/v1/connectionProfiles/search/TDMSearch] c.c.t.c.e.h.GlobalExceptionConvertor: HTTP Error - INTERNAL SERVER ERROR: Exception: java.io.IOException: An established connection was aborted by the software in your host machine

java.io.IOException: An established connection was aborted by the software in your host machine


The main problem is detailed in the following lines:


o.h.e.j.s.SqlExceptionHelper: SQL Error: 983, SQLState: S0001
o.h.e.j.s.SqlExceptionHelper: Unable to access availability database 'GTREP_PD_Shadow' because the database replica is not in the PRIMARY or SECONDARY role. Connections to an availability database is permitted only when the database replica is in the PRIMARY or SECONDARY role. Try the operation again later. ClientConnectionId:a226bff6-c770-46cf-860f-70cd177de58b

c.c.t.c.e.h.GlobalExceptionConvertor: HTTP Error - INTERNAL SERVER ERROR: Exception: Unable to acquire JDBC Connection; SQL [n/a]; nested exception is org.hibernate.exception.SQLGrammarException: Unable to acquire JDBC Connection

com.microsoft.sqlserver.jdbc.SQLServerException: Unable to access availability database 'GTREP_PD_Shadow' because the database replica is not in the PRIMARY or SECONDARY role. Connections to an availability database is permitted only when the database replica is in the PRIMARY or SECONDARY role. Try the operation again later. ClientConnectionId:a226bff6-c770-46cf-860f-70cd177de58b



Please make sure that the database "GTREP_PD_SHADOW' is available and in the correct role to be accessed.