We attempted to upgrade APM10.5.2 Enterprise Manager cluster to 10.7 SP3 and we ran into trouble. Our environment has 1 MOM and 2 Collectors. The EM collector that is having issue is running the DB Cleanup Service, Stats Aggregation Service, and TIM Collection Service. The other EM collector is running good.
EM collector IntroscopeEnterpriseManager.log is showing the following WARN and ERROR:
1/10/21 03:30:04.392 AM EST [WARN] [ServicesCheck.Thread1] [Manager.com.timestock.tess.services.service.CheckServices] EM earlier was not shutdown properly, TIM.Collection.Svc service is trying to recover...
1/10/21 03:30:14.958 AM EST [ERROR] [ServicesCheck.Thread1] [Manager] Uncaught Exception in Enterprise Manager: In thread 'ServicesCheck.Thread1' and the message is: java.lang.StackOverflowError
Release : 10.7.0
Component : APMISP
The problem is related to the amount of data that is pending, and needs to be processed by the EM collector for CEM.
We have made the following suggestions in this type of issue. You only need to take these steps (one or more) if you continue to see this exact same issue in the future.
Increase stack size -Xss from 512K to 1024K (1MB) (JVM option in APM-Home\Introscope_Enterprise_Manager.lax) if EM is running as process or
(JVM option in APM-Home\bin\EMService.conf) if EM is running as service (on windows).
Increase the heap size and set both -Xms and -Xmx same value. (JVM option in Introscope_Enterprise_Manager.lax) if EM is running as process or
if EM is running as service (on windows) (in APM-Home\bin\EMService.conf).
# Initial Java Heap Size (in MB)
# Maximum Java Heap Size (in MB)
Make sure you set same value for both, like 8192 (8GB)
Information is given in TechDoc: https://ca-broadcom.wolkenservicedesk.com/external/article?articleId=48602
Also find additional information in online guide.