Connectivity check failed: Deployer on host <agentname> does not respond
Article ID: 205465
Updated On:
Products
Issue/Introduction
While running a deployment the following errors were returned:
- Error: <Process Name> has a process error - Connectivity check failed
- Error: <Process Name> has a process error - Deployer on host <Agent hostname> does not respond.
- Error: <Process Name> has a process error - Server <Agent hostname>(Agent IPAddress) error
Cause
When an Execution Server starts a job on an agent, it first tries to PING the agent (see Additional Information for log msgs). The errors above indicate that the Execution Server sent the PING message to the Agent but did not get a response.
Environment
Release : 6.6
Component : CA RELEASE AUTOMATION CORE
Resolution
There are two options to resolve this problem:
- Resolve connectivity issues between the NES and Agent.
- Connect the agent to a different NES (Execution Server).
Additional Information
Troubleshooting
If the PING message sent by the NES is not received by the agent then it is expected that the underlying problem is a network issue. If additional analysis is needed then the following should be gathered after the problem has been reproduced:
- TCPDump/Wireshark trace started on the NES and NAG experiencing this problem. Having the traces started on both offers the best amount of detail. At a minimum, the network trace should be captured on the NES.
- The logs folder from the NES and NAG. This way we'll understand what the app believes has happened during the timeframe in question.
When this type of error occurs, consider using the NES's JMX to manually PING the agent machine in question. This will give you the capability to reproduce problems sending messages without starting deployments. If you can reproduce the message sending problem then it can give an opportunity to get the network tracing enabled, reproduce, capture data. To do this:
-
- Open JMX for the Execution Server managing the agent in question.
- From the JMX find: com.nolio.nimi.jmx:name=nimiJMX,type=NimiJMX bean/page.
- Then, use the sendMessageTo method. To use this method, there is a message (p2 field) to an agent node_id (p1 field).
- Fill out Agent’s NodeID in the p1 field; and
- Fill out a message in the p2 field (ex: WhereAreYou).
- Then, click invoke.
If it gets a response then it will reflect that the message was sent. If it doesn't get a response then it will eventually give a timeout error.
Example Messages
See below for example message that you should be able to see exchanged between the NES and Agent upon a successful PING.
Note:
- These messages can be found in the nimi.log
- You can use the ID in the payload to track the message between nodes.
Execution Server:
2020-12-07 15:09:50,397 [JobExecutorThread-6] DEBUG (com.nolio.nimi.appmsg.durability.DurableCommunicationApi:144) - Got new message: es_<nes_nodeId>_160693912061731:payload=[ID:[email protected]_<nes_nodeId>, from:es_<nes_nodeId>, to:[email protected]<agent_nodeId>- PING]
Agent Server:
2020-12-07 15:09:50,503 [New I/O server worker #1-2] DEBUG (com.nolio.nimi.appmsg.durability.DurableCommunicationApi:233) - Received shipping: es_<nes_nodeId>_160693912061731:payload=[ID:[email protected]_<nes_nodeId>, from:es_<nes_nodeId>, to:[email protected]<agent_nodeId>- PING]
Agent Server:
2020-12-07 15:09:50,506 [Communication Msg Processor-2] DEBUG (com.nolio.nimi.appmsg.durability.DurableCommunicationApi:155) - Got new message: <agent_nodeId>_160737107556804:payload=[ID:[email protected]<agent_nodeId>, from:<agent_nodeId>, to:[email protected]_<nes_nodeId>- [Response for message: [email protected]_<nes_nodeId>]]
Execution Server:
2020-12-07 15:09:50,509 [New I/O client worker #1-1] DEBUG (com.nolio.nimi.appmsg.durability.DurableCommunicationApi:222) - Received shipping: <agent_nodeId>_160737107556804:payload=[ID:[email protected]<agent_nodeId>, from:<agent_nodeId>, to:[email protected]_<nes_nodeId>- [Response for message: [email protected]_<nes_nodeId>]]
Feedback
