MOM fails to restart and the log shows that after the SmartStor Subsystem starts up the only message is "The EM failed to start. 2244"

Products

• APP PERF MANAGEMENT
• CA Application Performance Management Agent (APM / Wily / Introscope)
• CUSTOMER EXPERIENCE MANAGER
• INTROSCOPE

Issue/Introduction

MOM fails to restart and the IntroscopeEnntepriseManager.log shows:

[INFO] [main] [Manager] Using data directory: /opt/apm/smartstor 

[INFO] [main] [Manager] Available processors is 4 

[ERROR] [main] [Manager] The EM failed to start. 2244 

[INFO][main] [Manager] Shutting down the Isengard server

[INFO] [main] [Manager] Orderly shutdown complete.

Cause

After enabling DEBUG in the MOM IntrocopeEnterpriseManager.properties file an exception is then visible:

[ERROR] [main] [Manager] The EM failed to start. 2244 

[VERBOSE] [main] [Manager] 

 

java.lang.ArrayIndexOutOfBoundsException: 2244 

at com.wily.util.bit.BitPacking.readInt(BitPacking.java:345) 

at com.wily.introscope.spec.server.beans.metricdata.QueryFileDataBlob.getRange(QueryFileDataBlob.java:176) 

at com.wily.introscope.server.enterprise.entity.fsdb.query.SlowQueryFileMetricIDSetScanner$1.block(SlowQueryFileMetricIDSetScanner.java:60) 

at com.wily.introscope.server.enterprise.entity.fsdb.queryfile.QueryFile.scanByteBlocks(QueryFile.java:430) 

at com.wily.introscope.server.enterprise.entity.fsdb.query.SlowQueryFileMetricIDSetScanner.scanMetricIDs(SlowQueryFileMetricIDSetScanner.java:53) 

... 

The symptoms indicate a possible smartstor related corruption but no parsing error for a smartstor file is present in the log.

 

Environment

Release: CEMUGD00200-9.7-Introscope to CA Application-Performance Management-Upgrade Main
Component:

Resolution

The latest timestamp file in the smartstor directory is a .spool file from the last successful/orderly shutdown.  Every hour the EM will process the .spool file to a data file and also on EM startup if a spool file still exists from previous shutdown.

On the presumption that the MOM was failing when processing the .spool file on startup, the file was backed up and removed. That allowed the MOM to start successfully. 

Additional Information

• The loss of the spool file means just losing the data between the timestamp of the last .data file and the timestamp of the spool file.
• Also for a MOM its smartstor contains no real agent metrics i.e. it only contains supportability metrics about the MOM itself or metrics generated directly on the MOM itself (e.g. virtual agents in agentclusters.xml).