SOI manager hangs and we see an error in the logs

book

Article ID: 206274

calendar_today

Updated On:

Products

CA Service Operations Insight (SOI)

Issue/Introduction

The SOI manager will hang and alerts are not being processed in a timely manner.

When this occurs, we see this error in the soimgr.log

2021-01-05 10:17:09,743 INFO  [PoolThread-156: AlarmRepoPool => AlarmRepositoryTask -4503599642302583] ?.?(?)  - DefaultDBTableManager: error reading table class com.ca.sam.common.escalation.EscalationPolicyLink:

org.hibernate.HibernateException: The internal connection pool has reached its maximum size and no connection is currently available!

 

The processing does eventually catch up but it takes several hours.

Cause

If the SOI manager, the SOI SQL server and/or the network are slow, we could see a delay in processing individual SQL transactions, resulting in the exhaustion of the SQL connection pool.

Environment

Release : 4.2

Component : Service Operations Insight (SOI) Manager

Resolution

The connection pool is controlled by this file:

C:\Program Files (x86)\CA\SOI\tomcat\lib\hibernate.cfg.xml

<!-- JDBC connection pool (use the built-in) -->

    <property name="connection.pool_size">20</property>

 

You must restart the SOI manager services for this change to take effect.

 

Increase in increments of 5, we have not tested over 30.

Additional Information

Full stack:

2021-01-05 10:17:09,743 INFO  [PoolThread-156: AlarmRepoPool => AlarmRepositoryTask -4503599642302583] ?.?(?)  - DefaultDBTableManager: error reading table class com.ca.sam.common.escalation.EscalationPolicyLink:

org.hibernate.HibernateException: The internal connection pool has reached its maximum size and no connection is currently available!

            at org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl$PooledConnections.poll(DriverManagerConnectionProviderImpl.java:290)

            at org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl$PoolState.getConnection(DriverManagerConnectionProviderImpl.java:440)

            at org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl.getConnection(DriverManagerConnectionProviderImpl.java:171)

            at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:38)

            at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:108)

            at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:138)

            at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getConnectionForTransactionManagement(LogicalConnectionManagedImpl.java:273)

            at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:281)

            at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:246)

            at org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:83)

            at org.hibernate.internal.AbstractSharedSessionContract.beginTransaction(AbstractSharedSessionContract.java:471)

            at sun.reflect.GeneratedMethodAccessor130.invoke(Unknown Source)

            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

            at java.lang.reflect.Method.invoke(Method.java:498)

            at org.hibernate.context.internal.ThreadLocalSessionContext$TransactionProtectionWrapper.invoke(ThreadLocalSessionContext.java:351)

            at com.sun.proxy.$Proxy55.beginTransaction(Unknown Source)

            at com.ca.sam.manager.hibernate.HibernateUtil.beginTransaction(HibernateUtil.java:775)

            at com.ca.sam.manager.hibernate.HibernateUtil.beginTransaction(HibernateUtil.java:761)

            at com.ca.sam.manager.escalation.EscalationRelTableHandler.readModelRelations(EscalationRelTableHandler.java:343)

            at com.ca.sam.manager.relation.SamRelationRepository.readModelRelations(SamRelationRepository.java:390)

            at com.aprisma.spectrum.app.topo.web.relation.RelationRepository.readModelRelations(RelationRepository.java:304)

            at com.aprisma.spectrum.app.topo.web.relation.RelationRepository.readAssociations(RelationRepository.java:278)

            at com.aprisma.spectrum.app.topo.common.relation.AbstractRelationRepository.getAssociations(AbstractRelationRepository.java:741)

            at com.aprisma.spectrum.app.topo.common.relation.AbstractRelationRepository.getAssociations(AbstractRelationRepository.java:432)

            at com.aprisma.spectrum.app.topo.common.relation.AbstractRelationRepository.getAssociations(AbstractRelationRepository.java:403)

            at com.aprisma.spectrum.app.topo.common.relation.AbstractRelationRepository.getAssociations(AbstractRelationRepository.java:354)

            at com.ca.sam.manager.escalation.AlarmEscalationService.checkPolicies(AlarmEscalationService.java:1885)

            at com.ca.sam.manager.escalation.AlarmEscalationService.alarmChanged(AlarmEscalationService.java:3055)

            at com.ca.sam.manager.alarm.AlarmRepository$7$1.performTask(AlarmRepository.java:7707)