Error building data: PxSystemException:Error building data for matching events J2CA1010E: Connection not available; timed out waiting for 10 seconds.

Products

CA Identity Manager

Issue/Introduction

DB Connection timeouts intermittently cause tasks to fail.

We see many of the following errors which seem to always be followed by "PxSystemException:Error building data for matching events "
J2CA1010E: Connection not available; timed out waiting for 10 seconds.
at com.netegrity.llsdk6.imsimpl.jdbcmanagedobject.JDBCManagedObjectProvider.find(JDBCManagedObjectProvider.java:655)
at com.netegrity.llsdk6.imsimpl.jdbcmanagedobject.JDBCManagedObjectProvider.find(JDBCManagedObjectProvider.java:752)
at com.netegrity.llsdk6.imsimpl.jdbcmanagedobject.JDBCManagedObjectProvider.find(JDBCManagedObjectProvider.java:547)
at com.ca.identitymanager.policyxpress.engine.PxProvider.getPoliciesForEvent(PxProvider.java:93)

Environment

Identity Manager 14.3, 14.4, 14.5

Application Server: IBM WebSphere 8.5.5.17

Application: IDM 14.3 CP2

Database: SQL server 2017

Cause

DB Connection timeouts intermittently cause tasks to fail.

IBM WebSphere Support advises that IM is not closing connections properly and is causing the connection pool to become exhausted.

We see many of the following errors which seem to always be followed by "PxSystemException:Error building data for matching events "
J2CA1010E: Connection not available; timed out waiting for 10 seconds.
at com.netegrity.llsdk6.imsimpl.jdbcmanagedobject.JDBCManagedObjectProvider.find(JDBCManagedObjectProvider.java:655)
at com.netegrity.llsdk6.imsimpl.jdbcmanagedobject.JDBCManagedObjectProvider.find(JDBCManagedObjectProvider.java:752)
at com.netegrity.llsdk6.imsimpl.jdbcmanagedobject.JDBCManagedObjectProvider.find(JDBCManagedObjectProvider.java:547)
at com.ca.identitymanager.policyxpress.engine.PxProvider.getPoliciesForEvent(PxProvider.java:93)
... 23 more
Caused by: com.ibm.websphere.ce.j2c.ConnectionWaitTimeoutException: J2CA1010E: Connection not available; timed out waiting for 10 seconds.
at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java:1782)
at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3907)
at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3085)
at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:1554)
at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:1033)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:644)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:611)
at com.netegrity.llsdk6.imsimpl.jdbcmanagedobject.JDBCManagedObjectProvider.find(JDBCManagedObjectProvider.java:557)
... 26 more

[11/11/20 12:40:28:782 PST] 000000f0 FreePool E J2CA0045E: Connection not available while invoking method createOrWaitForConnection for resource iam/im/jdbc/jdbc/objectstore....

IBM WAS is reporting "Connection reset by peer" intermittently for connection to the object store DB.

One of the Xpress Policies executes a stored procedure. The PX includes just a simple sql statement--nothing unusual. We increased DB pool settings from max 200 to 350, which only increases the number of connection and task failures as the connections do not get returned to the pool.

IBM advises that we need to close all connections by invoking connection.close() method, we already perform. They also recommend using an "unshareable" DB connection setting,

How to change from Shareable to Unshareable Connections

Resolution

Implement IBM WebSphere Support's recommendation to make the db connection "unshareable."

https://www.ibm.com/support/pages/node/1080489?lang=en

From the IBM Administrative Console , navigate to the following path:

Resource >JDBC > Data sources > < IDM datasource (ex. objectstore)> Connection pool properties > Connection pool custom properties

set
Name: defaultConnectionTypeOverride
Value: unshare