Audit DB Error: Is Dead or invalid credentials

Products

• SITEMINDER
• CA Single Sign On Agents (SiteMinder)
• CA Single Sign On Federation (SiteMinder)
• CA Single Sign On Secure Proxy Server (SiteMinder)
• CA Single Sign On SOA Security Manager (SiteMinder)

Issue/Introduction

Customer conducts failover to point policy server from one data center to another, a small number of policy servers throw this DB error post failover. 

Sometimes it can be 5 minutes later, sometimes 15 minutes later.

There was no network or firewall outage during this time.  Nothing happened to the DB during the time frame.  There are many other policy servers that did not have this problem in the same environment.

The policy server is unable to re-establish connection to DB until PS is restarted.

Policy serer is pinging <AuditDB> every 15 second as expected, however, once the DB connection is marked as DEAD, the dead connection was never cleaned up.

The policy server trace log continues to show thread [32555][140655538710272] lasted for hours, repeated with the exact same error.

[05/19/2023][05:43:38.212][05:43:38][32555][140655538710272][CSmDbUtilities.cpp:603][CSmDbMonitoredClass::MonitorThreadProc][][][][][][][][][][][][][][][][][][][][][a connection to 'ODBC' server '<AuditDB>'. Is Dead or invalid credentials.]
[05/19/2023][05:43:38.226][05:43:38][32555][140655374755584][CSmDbUtilities.cpp:760][CSmDbMonitoredClass::KillMonitor][][][][][][][][][][][][][][][][][][][][][Killing Monitor thread]

[05/19/2023][05:43:38.226][05:43:38][32555][140655374755584][CSmDbODBC.cpp:1573][CSmDbConnectionODBC::Disconnect][][][][][][][][][][][][][][][][][][][][][Connection <AuditDB>: Disconnecting the connection.]

...

[05/19/2023][05:43:53.227][05:43:53][32555][140655538710272][CSmDbUtilities.cpp:603][CSmDbMonitoredClass::MonitorThreadProc][][][][][][][][][][][][][][][][][][][][][a connection to 'ODBC' server '<AuditDB>'. Is Dead or invalid credentials.]

...

[05/19/2023][13:48:22.036][13:48:22][32555][140655538710272][CSmDbUtilities.cpp:603][CSmDbMonitoredClass::MonitorThreadProc][][][][][][][][][][][][][][][][][][][][][a connection to 'ODBC' server '<AuditDB>'. Is Dead or invalid credentials.]

Environment

Release : 12.8.05

Policy Server O/S: RHEL7

<AuditDB> server: Oracle 18.0.0.0.0

Cause

The particular DB Monitor thread was NOT properly closed as it should be.

Resolution

Broadcom engineering can provide policy server fix Linux binary libsmutilities.so for version 12.8.05.

If customer upgrades to 12.8.07, then it will not exhibit the problem.

Additional Information

DE566959