Unable to start Enterprise manager on Collector

Products

• CA Application Performance Management Agent (APM / Wily / Introscope)
• CA Application Performance Management (APM / Wily / Introscope)
• INTROSCOPE
• DX Application Performance Management

Issue/Introduction

seeing multiple errors in log when i attempt to start the service. Here are the logs I am seeing.

[ERROR] [Smartstor/Superstor Spool] [Manager] Can't write timeslice 1604056170000 to spool: java.io.SyncFailedException: sync failed
[ERROR] [Smartstor/Superstor Spool] [Manager] Can't write timeslice 1604056185000 to spool: java.io.SyncFailedException: sync failed
[ERROR] [master clock] [Manager] Shutting down the EM because of disk I/O failure in 5 continuous timeslices.
[ERROR] [master clock] [Manager] Shutting down the EM because of disk I/O failure in 5 continuous timeslices.
[WARN] [master clock] [Manager.Clock] Timeslice processing delayed due to system activity. Combining data from timeslices 106937084 to 106937085
[ERROR] [master clock] [Manager] Shutting down the EM because of disk I/O failure in 5 continuous timeslices.
[ERROR] [master clock] [Manager] Shutting down the EM because of disk I/O failure in 5 continuous timeslices.
[ERROR] [master clock] [Manager] Shutting down the EM because of disk I/O failure in 5 continuous timeslices.
[WARN] [master clock] [Manager.Clock] Timeslice processing delayed due to system activity. Combining data from timeslices 106937088 to 106937091
[ERROR] [master clock] [Manager] Shutting down the EM because of disk I/O failure in 5 continuous timeslices.
[ERROR] [master clock] [Manager] Shutting down the EM because of disk I/O failure in 5 continuous timeslices.
[ERROR] [master clock] [Manager] Shutting down the EM because of disk I/O failure in 5 continuous timeslices.
[INFO] [Thread-30604] [Manager] Beginning orderly shutdown...
[ERROR] [master clock] [Manager] Shutting down the EM because of disk I/O failure in 5 continuous timeslices.
[ERROR] [master clock] [Manager] Shutting down the EM because of disk I/O failure in 5 continuous timeslices.
[WARN] [pool-23-thread-1] [Manager.com.wily.apm.tess.isengard.EMBizRecordingServiceBean] Could not persist new agent : SuperDomain|jtcimweb2|.NET Process|Public%1 due to Could not open Hibernate Session for transaction; nested exception is org.hibernate.exception.GenericJDBCException: Cannot open connection
[WARN] [pool-23-thread-1] [Manager.com.wily.apm.tess.isengard.EMBizRecordingServiceBean] Could not persist new agent : SuperDomain|jtcimweb1|.NET Process|Public%1 due to Could not open Hibernate Session for transaction; nested exception is org.hibernate.exception.GenericJDBCException: Cannot open connection
[ERROR] [Perst Baseline insertion] [Manager] Uncaught Exception in Enterprise Manager: In thread 'Perst Baseline insertion' and the message is: org.garret.perst.AssertionFailed: Assertion failed

org.garret.perst.AssertionFailed: Assertion failed
        at org.garret.perst.Assert.that(Assert.java:18)
        at org.garret.perst.impl.PagePool.find(PagePool.java:93)
        at org.garret.perst.impl.PagePool.copy(PagePool.java:122)
        at org.garret.perst.impl.StorageImpl.allocate(StorageImpl.java:413)
        at org.garret.perst.impl.StorageImpl.cloneBitmap(StorageImpl.java:773)
        at org.garret.perst.impl.StorageImpl.storeObject0(StorageImpl.java:2710)
        at org.garret.perst.impl.StorageImpl.storeObject(StorageImpl.java:2604)
        at com.wily.introscope.server.enterprise.etc.WilyPersistent.store(WilyPersistent.java:56)
        at org.garret.perst.impl.StorageImpl.endThreadTransaction(StorageImpl.java:2267)
        at org.garret.perst.impl.StorageImpl.endThreadTransaction(StorageImpl.java:2252)
        at com.ca.apm.baseline.engine.core.PerstManager.bulkInsert(PerstManager.java:222)
        at com.ca.apm.baseline.engine.BaselineDataManager.saveBaselinesToPerst(BaselineDataManager.java:539)
        at com.ca.apm.baseline.engine.BaselineDataManager.access$4(BaselineDataManager.java:523)
        at com.ca.apm.baseline.engine.BaselineDataManager$PerstInserter.run(BaselineDataManager.java:633)

Environment

Release : 10.7.0

Component : APM Agents

Cause

Smartstor metadata is corrupted due to an unclean shutdown. 

Please have the customer zip up their smartstor data directory including all subdirectories and upload it to the support case.  Support can then repair the metadata.

Resolution

Repaired customer's metadata and now both EM's are up and running.