MetricClamp/agent.error .limit /maximum disk usage on All Collectors and MOM

Products

CA Application Performance Management (APM / Wily / Introscope)

Issue/Introduction

Customer hitting various clamps

1) introscope.enterprise.agent.error.limit

2) introscope.enterprise.transactionevents.storage.max.disk.usage

3)introscopeagent.metricClamp

4)introscope.enterprisemanager.agent.metric.limit

Environment

Release : 10.7.0

Component : Integration with APM

Resolution

Customer is hitting clamps. This will stop creating metrics.

You have two choices
1) Send less metrics from the agents impacted.
2) Increase the clamp. But you may hit it again

Good overall resource -- https://broadcom-stage.adobecqms.net/us/en/ca-enterprise-software/it-operations-management/application-performance-management/10-7/administrating/properties-files-reference/apm-events-thresholds-config-xml.html

Clamps:

1) introscope.enterprise.agent.error.limit
Default is 10. Your value is 10. Set in EM_Home\config\apm-events-thresholds-config.xml

This seems like a reasonable value.
Next action: See what the agent error is and attempt to resolve. Increasing clamp won't make much of a difference since the error will continue.
2) introscope.enterprise.transactionevents.storage.max.disk.usage
The maximum desired disk usage in MB for the trace storage. If this maximum is exceeded,then the daily historical trace stores will be deleted starting with the oldest first until the total historical trace storage size is below this value.

The current days trace store actively storing traces will not be deleted even if its size exceeds this property value. The size of the trace
index is not considered when determining what historical trace stores to delete
Default is 1024. Your value is 1024. Set in EM_Home\config\apm-events-thresholds-config.xml

Next action:

https://knowledge.broadcom.com/external/article?articleId=195285

1. increase the space for each collector and make the value synced up across all EMs. Restart of EM is required.

https://knowledge.broadcom.com/external/article?articleId=31622 Reduce traces from Agents who reports way too many traces

2. Reduce backend traces
https://knowledge.broadcom.com/external/article?articleId=5299

3)introscopeagent.metricClamp

Default is 50000. Your value is 50000. Set in IntroscopeAgent.profile

Next steps:
KB has lots of good suggestions:
https://knowledge.broadcom.com/external/article/201644/apm-console-showing-agent-metric-clamp.html

If you increase the clamp you need to make sure that APM cluster/Standalone EM does not have an high impact.

In the IntroscopeAgent.profile, you can adjust the following.

introscope.agent.metricClamp=5000

to

introscope.agent.metricClamp=8000

OR

introscope.agent.metricClamp=10000

You can test on 1 agent and see the results

2) Reduce agent metrics

4)introscope.enterprisemanager.agent.metrics.limit

Set this property with the value for which agent reporting is clamped when the value is exceeded.
For example, if the default number of metrics, 50,000, is met, then the following occur:
The agent reports only the 50,000 metrics.
The agent does not send new metrics to the Enterprise Manager

Default is 50000. Your value is 50000. Set in EM_Home\config\apm-events-thresholds-config.xml.

Reduce metrics . Can increase clamp but then may hit higher clamp limit.