Prod Cluster Down.....

Products

• CA Application Performance Management Agent (APM / Wily / Introscope)
• CA Application Performance Management (APM / Wily / Introscope)
• INTROSCOPE
• DX Application Performance Management

Issue/Introduction

Have a prod cluster that is down hard and will not start... MOM is fine.  Collectors will not start.... 

Example: IntroscopeEnterpriseManager.log states:

 [INFO] [main] [Manager] Available processors is 6

 [ERROR] [main] [Manager] The EM failed to start. com.wily.introscope.spec.metric.BadlyFormedNameException: "Backends|yyydb on xxx (DB2 DB)|SQL|Prepared|Query|" is not a legal Metric name.

 [INFO] [main] [Manager] Shutting down the Isengard server

 [INFO] [main] [Manager] Orderly shutdown complete.

******************************

Tried looking for the data with SmartstoreTools:

./SmartStorTools.sh test_regex -metrics ".*Backends|yyydb on xxx (DB2 DB)|SQL|.*" -src /CAAPMDATA/Introscope/data > out.txt

Provided the following error and stopped:

log4j:WARN No appenders could be found for logger (org.springframework.osgi.extender.internal.activator.ContextLoaderListener).

log4j:WARN Please initialize the log4j system properly.

log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

com.wily.introscope.server.enterprise.entity.meta.MetadataException: com.wily.introscope.spec.metric.BadlyFormedNameException: Metric_Name_Is_Badly_Formed

        at com.wily.introscope.server.enterprise.entity.meta.MetadataStoreSystem.getMatchingItems(MetadataStoreSystem.java:1127)

        at com.wily.introscope.em.smartstor.tools.fileapps.RegexTester.printMatchingMetrics(RegexTester.java:265)

        at com.wily.introscope.em.smartstor.tools.fileapps.RegexTester.main(RegexTester.java:148)

        at com.wily.introscope.em.smartstor.tools.fileapps.RegexTester.main(RegexTester.java:71)

        at com.wily.introscope.em.smartstor.tools.SmartStorTools.main(SmartStorTools.java:77)

        at com.wily.introscope.eclipseapp.SmartStorTools.start(SmartStorTools.java:11)

        at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:193)

        at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)

        at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)

        at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:386)

        at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:549)

        at org.eclipse.equinox.launcher.Main.basicRun(Main.java:504)

        at org.eclipse.equinox.launcher.Main.run(Main.java:1236)

        at org.eclipse.equinox.launcher.Main.main(Main.java:1212)

Caused by: com.wily.introscope.spec.metric.BadlyFormedNameException: Metric_Name_Is_Badly_Formed

        at com.wily.introscope.spec.metric.AgentMetricPrefixParser.validateSegment(AgentMetricPrefixParser.java:118)

        at com.wily.introscope.spec.metric.AgentMetricPrefixParser.validate(AgentMetricPrefixParser.java:104)

        at com.wily.introscope.spec.metric.AgentMetricPrefixParser.<init>(AgentMetricPrefixParser.java:33)

        at com.wily.introscope.spec.metric.AgentMetricPrefix.<init>(AgentMetricPrefix.java:176)

        at com.wily.introscope.spec.metric.AgentMetricPrefix.<init>(AgentMetricPrefix.java:174)

        at com.wily.introscope.spec.metric.AgentMetricPrefix$Factory.create(AgentMetricPrefix.java:48)

        at com.wily.introscope.server.enterprise.entity.meta.cache.AgentMetricPrefixCache.getOrAddToCache(AgentMetricPrefixCache.java:97)

        at com.wily.introscope.spec.metric.AgentMetricPrefix.getFromCache(AgentMetricPrefix.java:340)

        at com.wily.introscope.spec.metric.AgentMetricPrefix.getAgentMetricPrefix(AgentMetricPrefix.java:369)

        at com.wily.introscope.spec.metric.AgentMetric.<init>(AgentMetric.java:455)

        at com.wily.introscope.server.enterprise.entity.meta.cache.AgentMetricCache.lookupMetricWithTypeCheck(AgentMetricCache.java:105)

        at com.wily.introscope.spec.metric.AgentMetric.lookupMetric(AgentMetric.java:587)

        at com.wily.introscope.spec.metric.AgentMetric.getAgentMetricFromCache(AgentMetric.java:616)

        at com.wily.introscope.spec.metric.AgentMetric.getAgentMetric(AgentMetric.java:798)

        at com.wily.introscope.server.enterprise.entity.meta.table.TableTreeToMetric.getAgentMetric(TableTreeToMetric.java:233)

        at com.wily.introscope.server.enterprise.entity.meta.MetadataStoreSystem$14.data(MetadataStoreSystem.java:1104)

        at com.wily.introscope.server.enterprise.entity.rocksdb.RocksDBSystem.iterate(RocksDBSystem.java:157)

        at com.wily.introscope.server.enterprise.entity.rocksdb.RocksDBSystem.iterate(RocksDBSystem.java:131)

        at com.wily.introscope.server.enterprise.entity.meta.MetadataStoreSystem.getMatchingItems(MetadataStoreSystem.java:1088)

        ... 18 more

 

Cause

Agent creating badly formed metric

Environment

Release : 10.7.0

Component : APM Agents

Resolution

Engineering fixed up the corrupted Metadata. They also provided a jar file that should fix this metadata corruption from happening in the future.