MOM fails to restart and the IntroscopeEnntepriseManager.log shows:
[INFO] [main] [Manager] Using data directory: /opt/apm/smartstor
[INFO] [main] [Manager] Available processors is 4
[ERROR] [main] [Manager] The EM failed to start. 2244
[INFO][main] [Manager] Shutting down the Isengard server
[INFO] [main] [Manager] Orderly shutdown complete.
After enabling DEBUG in the MOM IntrocopeEnterpriseManager.properties file an exception is then visible:
[ERROR] [main] [Manager] The EM failed to start. 2244
[VERBOSE] [main] [Manager]
java.lang.ArrayIndexOutOfBoundsException: 2244
at com.wily.util.bit.BitPacking.readInt(BitPacking.java:345)
at com.wily.introscope.spec.server.beans.metricdata.QueryFileDataBlob.getRange(QueryFileDataBlob.java:176)
at com.wily.introscope.server.enterprise.entity.fsdb.query.SlowQueryFileMetricIDSetScanner$1.block(SlowQueryFileMetricIDSetScanner.java:60)
at com.wily.introscope.server.enterprise.entity.fsdb.queryfile.QueryFile.scanByteBlocks(QueryFile.java:430)
at com.wily.introscope.server.enterprise.entity.fsdb.query.SlowQueryFileMetricIDSetScanner.scanMetricIDs(SlowQueryFileMetricIDSetScanner.java:53)
...
The symptoms indicate a possible smartstor related corruption but no parsing error for a smartstor file is present in the log.
The latest timestamp file in the smartstor directory is a .spool file from the last successful/orderly shutdown. Every hour the EM will process the .spool file to a data file and also on EM startup if a spool file still exists from previous shutdown.
On the presumption that the MOM was failing when processing the .spool file on startup, the file was backed up and removed. That allowed the MOM to start successfully.