Use of the SmartStor merge tool on APM 10.5.x data can potentially corrupt SmartStor data.

Products

• CA Application Performance Management Agent (APM / Wily / Introscope)
• INTROSCOPE

Issue/Introduction

Seeing in the APM 10.5.x EM logs many messages such as 

The log is filled with messages like: 

8/30/17 09:32:45.775 PM EST [WARN] [Smartstor/Superstor Spool] [Manager] Metric key id 0x100000029 illegally used by two metrics: [SuperDomain|Custom Metric Host (Virtual)|Custom Metric Process (Virtual)|Custom Metric Agent (Virtual) (Collector2)Enterprise Manager|Configuration:Number of Agent Clusters] and [SuperDomain|Custom Metric Host (Virtual)|Custom Metric Process (Virtual)|Custom Metric Agent (Virtual) (Collector2)Alerts|Default:Backend Errors]; ignoring second. 

 

Cause

This may be due to the Smartstor merge tool that is included in APM 10.5, corrupting the Smartstor metadata.

Environment

Upgrading to APM 10.5.x

Resolution

The recommendation from Engineering is try avoid using the merge tool if possible. If it already occurred, use the new SmartStor repair tool added in 10.5.1HF#39 or 10.5.2SP2 to repair the corrupted data. (repair_metadata, repair_data).

 

Additional Information

The same error can also happen with duplicate management modules -- After upgrade from 10.3 to 10.5 the MOM log is full of WARNing messages "Metric key id 0xnnnnnnnn illegally used by two metrics ...." which are repeated every 15 seconds  TEC1455096.