Customer conducts failover to point policy server from one data center to another, a small number of policy servers throw this DB error post failover.
Sometimes it can be 5 minutes later, sometimes 15 minutes later.
There was no network or firewall outage during this time. Nothing happened to the DB during the time frame. There are many other policy servers that did not have this problem in the same environment.
The policy server is unable to re-establish connection to DB until PS is restarted.
Policy serer is pinging <AuditDB> every 15 second as expected, however, once the DB connection is marked as DEAD, the dead connection was never cleaned up.
The policy server trace log continues to show thread [32555][140655538710272] lasted for hours, repeated with the exact same error.
[05/19/2023][05:43:38.212][05:43:38][32555][140655538710272][CSmDbUtilities.cpp:603][CSmDbMonitoredClass::MonitorThreadProc][][][][][][][][][][][][][][][][][][][][][a connection to 'ODBC' server '<AuditDB>'. Is Dead or invalid credentials.]
[05/19/2023][05:43:38.226][05:43:38][32555][140655374755584][CSmDbUtilities.cpp:760][CSmDbMonitoredClass::KillMonitor][][][][][][][][][][][][][][][][][][][][][Killing Monitor thread]
[05/19/2023][05:43:38.226][05:43:38][32555][140655374755584][CSmDbODBC.cpp:1573][CSmDbConnectionODBC::Disconnect][][][][][][][][][][][][][][][][][][][][][Connection <AuditDB>: Disconnecting the connection.]
...
[05/19/2023][05:43:53.227][05:43:53][32555][140655538710272][CSmDbUtilities.cpp:603][CSmDbMonitoredClass::MonitorThreadProc][][][][][][][][][][][][][][][][][][][][][a connection to 'ODBC' server '<AuditDB>'. Is Dead or invalid credentials.]
...
[05/19/2023][13:48:22.036][13:48:22][32555][140655538710272][CSmDbUtilities.cpp:603][CSmDbMonitoredClass::MonitorThreadProc][][][][][][][][][][][][][][][][][][][][][a connection to 'ODBC' server '<AuditDB>'. Is Dead or invalid credentials.]
Release : 12.8.05
Policy Server O/S: RHEL7
<AuditDB> server: Oracle 18.0.0.0.0
The particular DB Monitor thread was NOT properly closed as it should be.
DE566959