APM performance due to webservice calls - 500 Response for SOAP web service calls under high concurrent load due to a ConcurrentModificationException

Products

• CA Application Performance Management Agent (APM / Wily / Introscope)
• CA Application Performance Management (APM / Wily / Introscope)
• INTROSCOPE
• DX Application Performance Management

Issue/Introduction

We are heavily utilizing the SOAP metric data web service (https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/it-operations-management/application-performance-management/10-7/api-reference/ca-introscope-web-services-api-reference/polling-web-services/metrics-data-web-service.html) to fetch metric data from APM (using the operation “getMetricData”) using grafana which is integrated with APM. 

Sometimes the Enterprise Manager returns an error response with HTTP error code 500. In these cases the response always contains the following response body:

<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <soapenv:Body>
        <soapenv:Fault>
            <faultcode>soapenv:Server.userException</faultcode>
            <faultstring>java.util.ConcurrentModificationException</faultstring>
            <detail>
                <ns1:hostname xmlns:ns1="http://xml.apache.org/axis/">f22cb603e4a8</ns1:hostname>
            </detail>
        </soapenv:Fault>
    </soapenv:Body>
</soapenv:Envelope>

Environment

Introscope Enterprise Manager 10.7.x

Cause

This is a known issue with the Axis version bundled with the Introscope Enteprise Manager:
https://issues-test.apache.org/jira/browse/AXIS-2909

DE449168 - 20320127-HTTP 500 error codes from metrics data webservice (Axis 1.4.1120-1)

Resolution

Fixed in 10.7 HF#61

APM 10.7 Hotfixes
https://knowledge.broadcom.com/external/article/105898/apm-107-hotfixes.html