Agent data gaps in the APM metric view and too many alerts

Products

• DX Application Performance Management

Issue/Introduction

MQ admin reported too many alerts. It seems like agent is connecting and disconnect.  Alert settings look good. 

Metric View

 

We seeing these messages in agent log

4/11/22 12:17:58 AM EDT [INFO] [IntroscopeAgent.IsengardMessaging] WebSocket handshake complete.
4/11/22 12:17:58 AM EDT [INFO] [IntroscopeAgent.IsengardMessaging] WebSocket Client closed by server with code=1000, Bye
4/11/22 12:17:58 AM EDT [INFO] [IntroscopeAgent.IsengardMessaging] Isengard Input Stream closed.
4/11/22 12:17:58 AM EDT [WARN] [IntroscopeAgent.ConnectionThread] Failed to re-connect to the Introscope Enterprise Manager at 100.XXX.XXX.103:8081,com.wily.isengard.client.transport.websocket.WebSocketFactory (java.io.EOFException).
4/11/22 12:17:58 AM EDT [INFO] [IntroscopeAgent.ConnectionThread] The Agent will continue to attempt to re-connect to Introscope Enterprise Manager. Further failures will not be logged.
4/11/22 12:18:08 AM EDT [INFO] [IntroscopeAgent.IsengardMessaging] WebSocket handshake complete.

 

Environment

Release : 21.3

Component : Introscope

Cause

The alarms are result of disconnects in the network connection between Cloud Proxy and the MQ Server APMIA agent.

Resolution

Based on the analysis of the network trace, we find the following.

The alarms are result of disconnects in the network connection between Cloud Proxy and the MQ Server (APMIA agents).

Recommendations:
Have the network team look at network around the Cloud Proxy and correct the situation. Have the network team run their own analysis around the Cloud Proxy server.

Move the Cloud Proxy to another location where Cloud Proxy is not impacted by network issues.

Allow the Agents to connect directly to DXS APM instead of going through Cloud Proxy, assuming good network connection from agent to Internet(DXS). This would require reviewing your firewall configurations or put in proxy server connection at agent level or similar method to allow direct connections.