Agents continuously connecting/disconnecting to collectors (in Docker Containers)
search cancel

Agents continuously connecting/disconnecting to collectors (in Docker Containers)

book

Article ID: 221994

calendar_today

Updated On:

Products

CA Application Performance Management (APM / Wily / Introscope)

Issue/Introduction

we have a APM cluster (1 MOM/10 Collectors/2 webview) that is used to monitored are Kubernetes clusters are are seeing agents disconnecting and re-connecting within a few milliseconds.  Example on a single collector over a 12 hour period,  there was a total of 88969 disconnects/connections. We have a mix of agent version from 10.5.2 through 20.6.x.  the APM cluster running 10.7.0.308 (Build 997100)

Environment

Release : 10.7.0

Component : Integration with APM

Cause

We saw this error in the Agent logs.

7/16/21 08:12:46 AM EDT [ERROR] [IntroscopeAgent.Agent] Failed to re-register IAgentRemoteConfigurationService on Agent name change event.
7/16/21 08:13:54 AM EDT [ERROR] [IntroscopeAgent.Agent] Failed to re-register IAgentRemoteConfigurationService on Agent name change event.
https://ca-broadcomcsm.wolkenservicedesk.com/wolken/esd/knowledgebase_search?articleId=77019


7/16/21 03:40:10 AM EDT [ERROR] [IntroscopeAgent.Agent] 

com.wily.isengard.message.MessageUndeliverableException: closing MessageServiceClient for interface com.wily.introscope.spec.server.appmap.remotehttp.RemoteHttpCallService ; hashcode == 371715033
    at com.wily.isengard.messageprimitives.service.MessageServiceClient.sendRequest(MessageServiceClient.java:176)
    at com.wily.isengard.messageprimitives.service.MessageServiceClient.invoke(MessageServiceClient.java:359)
    at com.sun.proxy.$Proxy337.doPost(Unknown Source)
    at com.wily.introscope.spec.server.appmap.remotehttp.RemoteHttpCallClient$3.doCall(RemoteHttpCallClient.java:180)
    at com.wily.introscope.spec.server.appmap.remotehttp.RemoteHttpCallClient$Wrapper.call(RemoteHttpCallClient.java:105)
    at com.wily.introscope.spec.server.appmap.remotehttp.RemoteHttpCallClient.doPost(RemoteHttpCallClient.java:191)
    at com.wily.introscope.spec.server.appmap.remotehttp.RemoteHttpCallClient.doPost(RemoteHttpCallClient.java:170)
    at com.wily.introscope.agent.environment.AgentInformationReporter.reportEnvironmentInformationInner(AgentInformationReporter.java:235)
    at com.wily.introscope.agent.environment.AgentInformationReporter.reportEnvironmentInformation(AgentInformationReporter.java:55)
    at com.wily.introscope.agent.environment.EnvironmentAdministrator$11.execute(EnvironmentAdministrator.java:671)
    at com.wily.util.heartbeat.WaitAndExecuteBehavior.ITimestampedRunnable_execute(WaitAndExecuteBehavior.java:31)
    at com.wily.util.heartbeat.IntervalHeartbeat$BehaviorNode.execute(IntervalHeartbeat.java:972)
    at com.wily.util.heartbeat.IntervalHeartbeat.executeNextBehaviorAndCalculateSleepTime(IntervalHeartbeat.java:491)
    at com.wily.util.heartbeat.IntervalHeartbeat.access$2(IntervalHeartbeat.java:445)
    at com.wily.util.heartbeat.IntervalHeartbeat$HeartbeatRunnable.run(IntervalHeartbeat.java:668)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.RuntimeException: closing MessageServiceClient for interface com.wily.introscope.spec.server.appmap.remotehttp.RemoteHttpCallService ; hashcode == 371715033
    at com.wily.isengard.messageprimitives.service.MessageServiceClient.close(MessageServiceClient.java:109)
    at com.wily.isengard.postoffice.PostOffice.close(PostOffice.java:235)
    at com.wily.isengard.api.IsengardClient.cleanup(IsengardClient.java:251)
    at com.wily.isengard.api.IsengardClient.close(IsengardClient.java:237)
    at com.wily.introscope.agent.connection.IsengardClientConnection.tearDownConnection(IsengardClientConnection.java:79)
    at com.wily.introscope.agent.connection.IsengardServerConnectionManager.serverWentAway(IsengardServerConnectionManager.java:1066)
    at com.wily.introscope.agent.connection.ConnectionThread.attemptAdvancedConnection(ConnectionThread.java:443)
    at com.wily.introscope.agent.connection.ConnectionThread.connect(ConnectionThread.java:494)
    at com.wily.introscope.agent.connection.ConnectionThread.run(ConnectionThread.java:84)
    ... 1 more
    
    
7/16/21 08:09:26 AM EDT [WARN] [IntroscopeAgent.IsengardServerConnectionManager] Lost contact with the Introscope Enterprise Manager at localhost:5005.
7/16/21 08:09:26 AM EDT [INFO] [IntroscopeAgent.Agent] AutoTracingTriggerServiceBean passivated.
7/16/21 08:09:26 AM EDT [INFO] [IntroscopeAgent.Agent] AutoTracingTriggerServiceBean activated.
7/16/21 08:09:26 AM EDT [INFO] [IntroscopeAgent.Agent] ThreadDumpGenerator is enabled
7/16/21 08:09:26 AM EDT [INFO] [IntroscopeAgent.Instrumentation Manager] Agent Dynamic Instrumentation bean has been deployed.
7/16/21 08:09:26 AM EDT [WARN] [IntroscopeAgent.IsengardServerConnectionManager] Lost contact with the Introscope Enterprise Manager at localhost:5005.
7/16/21 08:09:26 AM EDT [INFO] [IntroscopeAgent.Agent] AutoTracingTriggerServiceBean passivated.
7/16/21 08:09:27 AM EDT [WARN] [IntroscopeAgent.Agent] Failed to re-register class com.wily.introscope.agent.beans.autotracing.AutoTracingTriggerServiceBean on Agent name change event.
7/16/21 08:09:27 AM EDT [WARN] [IntroscopeAgent.Thread dump] The ThreadDump Extension failed to register its service with the EM
7/16/21 08:09:27 AM EDT [WARN] [IntroscopeAgent.Agent] Failed to re-register class com.wily.introscope.autoprobe.dynamic.AgentDynamicInstrumentationBean on Agent name change event.
7/16/21 08:09:27 AM EDT [INFO] [IntroscopeAgent.Agent] AutoTracingTriggerServiceBean activated.
7/16/21 08:09:27 AM EDT [INFO] [IntroscopeAgent.Agent] ThreadDumpGenerator is enabled
7/16/21 08:09:27 AM EDT [INFO] [IntroscopeAgent.Instrumentation Manager] Agent Dynamic Instrumentation bean has been deployed.

Resolution

 

We recommended that the customer bypass the bootstrap agent and start the agent with the agent.jar in the wily folder.  This address the disconnection issues and error message.

Powered by Wolken Software