After upgrade to APM 10.7 EM collector continually shutting down

Products

• CA Application Performance Management (APM / Wily / Introscope)

Issue/Introduction

We attempted to upgrade APM10.5.2 Enterprise Manager cluster to 10.7 SP3 and we ran into trouble. Our environment has 1 MOM and 2 Collectors. The EM collector that is having issue is running the DB Cleanup Service, Stats Aggregation Service, and TIM Collection Service. The other EM collector is running good.

EM collector IntroscopeEnterpriseManager.log is showing the following WARN and ERROR:

1/10/21 03:30:04.392 AM EST [WARN] [ServicesCheck.Thread1] [Manager.com.timestock.tess.services.service.CheckServices] EM earlier was not shutdown properly, TIM.Collection.Svc service is trying to recover...

1/10/21 03:30:14.958 AM EST [ERROR] [ServicesCheck.Thread1] [Manager] Uncaught Exception in Enterprise Manager: In thread 'ServicesCheck.Thread1' and the message is: java.lang.StackOverflowError
java.lang.StackOverflowError
at org.hibernate.hql.ast.util.NodeTraverser.visitDepthFirst(NodeTraverser.java:41)

 

 

Environment

Release : 10.7.0

Component : APMISP

Resolution

The problem is related to the amount of data that is pending, and needs to be processed by the EM collector for CEM.

We have made the following suggestions in this type of issue. You only need to take these steps (one or more) if you continue to see this exact same issue in the future.

1-
Increase stack size -Xss from 512K to 1024K (1MB) (JVM option in APM-Home\Introscope_Enterprise_Manager.lax) if EM is running as process or
(JVM option in APM-Home\bin\EMService.conf) if EM is running as service (on windows).

2-
Increase the heap size and set both -Xms and -Xmx same value. (JVM option in Introscope_Enterprise_Manager.lax) if EM is running as process or
if EM is running as service (on windows) (in APM-Home\bin\EMService.conf).
# Initial Java Heap Size (in MB)
wrapper.java.initmemory=1024
# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=1024

Make sure you set same value for both, like 8192 (8GB)

3-
Set db.rowsFetchedPerQuery=1000
Information is given in TechDoc: https://ca-broadcom.wolkenservicedesk.com/external/article?articleId=48602
Also find additional information in online guide.
https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/application-performance-management/10-7/administrating/cem-configuration/the-default-tess-default-properties-file/modifying-the-tess-default-properties-property-default-file.html