Huge number of alarms due to JavaAgent Webservices metrics

Products

• DX SaaS
• DX Application Performance Management
• DX Operational Observability

Issue/Introduction

We are seeing a huge number of alarms getting generated,

 

Environment

DX Java Agent 24.9

Cause

The issue is due to defect #DE649418 incorrect normalization at the operationname causing many webservices metrics, below an example of unexpected metrics.

In this situation, an apm alert was created based on these metrics causing the many internal alerts.

Please note that these alerts don't show up in the alarm dashboard as these are not generated after the threshold crossing but are internal OK status events that are generated by APM as soon as a new metric is registered irrespective of threshold value. There events won't generate any notifications but due to flooding of these OK alerts the other critical/danger alerts are piled up in a queue.

Resolution

To be fixed in a next release Agent and PCF release. For an update contact Broadcom Support

WORKAROUND;

Option 1. On Agent side:

Disable tracing for the Spring Web Services Client and CXF Web Service Client, set the following property:

introscope.autoprobe.disable.tracergroups=SpringWSClientTemplateTracing,CxfWebServiceClientTracing.

This property can be applied either as a System Property when launching the application or by updating the IntroscopeAgent.profile file. 

 

Important: Impact of Disabling Tracing 

• • Metrics for Spring Web Services (SpringWS) and Apache CXF clients will no longer be visible in dashboards.
  • The client-side correlation essential for linking client and server calls for SOAP services will be disabled.

Note: This is a temporary measure until a permanent solution is available.

Option 2. On DXo2 UI

To prevent the multiple wanted internal alarms sent to DX OI, disable the APM alert that is using the "Webservices|Client" metrics