SOI manager hangs and we see an error in the logs

Products

• CA Service Operations Insight (SOI)

Issue/Introduction

The SOI manager will hang and alerts are not being processed in a timely manner.

When this occurs, we see this error in the soimgr.log

2021-01-05 10:17:09,743 INFO  [PoolThread-156: AlarmRepoPool => AlarmRepositoryTask -4503599642302583] ?.?(?)  - DefaultDBTableManager: error reading table class com.ca.sam.common.escalation.EscalationPolicyLink:

org.hibernate.HibernateException: The internal connection pool has reached its maximum size and no connection is currently available!

 

The processing does eventually catch up but it takes several hours.

Environment

Release : 4.2

Component : Service Operations Insight (SOI) Manager

Cause

If the SOI manager, the SOI SQL server and/or the network are slow, we could see a delay in processing individual SQL transactions, resulting in the exhaustion of the SQL connection pool.

Resolution

Before making any changes to SOI configurations, please be sure that the SQL host is well resourced and the network can handle the traffic.

 

The connection pool is controlled by this file:

C:\Program Files (x86)\CA\SOI\tomcat\lib\hibernate.cfg.xml

<!-- JDBC connection pool (use the built-in) -->

    <property name="connection.pool_size">20</property>

 

You must restart the SOI manager services for this change to take effect.

 

Increase in increments of 20, we have not tested over 100.

Additional Information

Full stack:

2021-01-05 10:17:09,743 INFO  [PoolThread-156: AlarmRepoPool => AlarmRepositoryTask -4503599642302583] ?.?(?)  - DefaultDBTableManager: error reading table class com.ca.sam.common.escalation.EscalationPolicyLink:

org.hibernate.HibernateException: The internal connection pool has reached its maximum size and no connection is currently available!

            at org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl$PooledConnections.poll(DriverManagerConnectionProviderImpl.java:290)

            at org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl$PoolState.getConnection(DriverManagerConnectionProviderImpl.java:440)

            at org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl.getConnection(DriverManagerConnectionProviderImpl.java:171)

            at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:38)

            at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:108)

            at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:138)

            at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getConnectionForTransactionManagement(LogicalConnectionManagedImpl.java:273)

            at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:281)

            at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:246)

            at org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:83)

            at org.hibernate.internal.AbstractSharedSessionContract.beginTransaction(AbstractSharedSessionContract.java:471)

            at sun.reflect.GeneratedMethodAccessor130.invoke(Unknown Source)

            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

            at java.lang.reflect.Method.invoke(Method.java:498)

            at org.hibernate.context.internal.ThreadLocalSessionContext$TransactionProtectionWrapper.invoke(ThreadLocalSessionContext.java:351)

            at com.sun.proxy.$Proxy55.beginTransaction(Unknown Source)

            at com.ca.sam.manager.hibernate.HibernateUtil.beginTransaction(HibernateUtil.java:775)

            at com.ca.sam.manager.hibernate.HibernateUtil.beginTransaction(HibernateUtil.java:761)

            at com.ca.sam.manager.escalation.EscalationRelTableHandler.readModelRelations(EscalationRelTableHandler.java:343)

            at com.ca.sam.manager.relation.SamRelationRepository.readModelRelations(SamRelationRepository.java:390)

            at com.aprisma.spectrum.app.topo.web.relation.RelationRepository.readModelRelations(RelationRepository.java:304)

            at com.aprisma.spectrum.app.topo.web.relation.RelationRepository.readAssociations(RelationRepository.java:278)

            at com.aprisma.spectrum.app.topo.common.relation.AbstractRelationRepository.getAssociations(AbstractRelationRepository.java:741)

            at com.aprisma.spectrum.app.topo.common.relation.AbstractRelationRepository.getAssociations(AbstractRelationRepository.java:432)

            at com.aprisma.spectrum.app.topo.common.relation.AbstractRelationRepository.getAssociations(AbstractRelationRepository.java:403)

            at com.aprisma.spectrum.app.topo.common.relation.AbstractRelationRepository.getAssociations(AbstractRelationRepository.java:354)

            at com.ca.sam.manager.escalation.AlarmEscalationService.checkPolicies(AlarmEscalationService.java:1885)

            at com.ca.sam.manager.escalation.AlarmEscalationService.alarmChanged(AlarmEscalationService.java:3055)

            at com.ca.sam.manager.alarm.AlarmRepository$7$1.performTask(AlarmRepository.java:7707)