Error: "Failed to ssh connect peer node 'NodeIP'", during VCHA configuration

Products

• VMware vCenter Server

Issue/Introduction

• Unable to configure vCenter Server HA.
• Configuration tab fails with error similar to:
  
  Failed to ssh connect peer node #.#.#.#


• In /var/log/vmware/vcha/sshConnect.log file, you see the error similar to:
  

  [YYYY:MM:DDTHH:MM:SS] INFO sshConnect Authentication (publickey) failed.
[YYYY:MM:DDTHH:MM:SS] WARNING sshConnect retry attempt 0: Authentication failed., not retrying
[YYYY:MM:DDTHH:MM:SS] INFO sshConnect Starting ssh connect to #.#.#.#
[YYYY:MM:DDTHH:MM:SS] INFO sshConnect Retry attempt 0
[YYYY:MM:DDTHH:MM:SS] INFO sshConnect Connected (version 2.0, client OpenSSH_7.1)
[YYYY:MM:DDTHH:MM:SS] INFO sshConnect Auth banner:
  

• In the /var/log/vmware/vpxd file, you see error similar to:
  
  [YYYY:MM:DDTHH:MM:SS] info vpxd[7FC49E0C1700] [Originator@6876 sub=Default opID=FlowBasedWizard-apply-20853-ngc-a3] [VpxLRO] -- ERROR task-145 -- FailoverClusterConfigurator -- vim.vcha.FailoverClusterConfigurator.configure: vmodl.fault.SystemError:
--> Result:
--> (vmodl.fault.SystemError) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>,
--> reason = "Failed to ssh connect peer node #.#.#.#"
--> msg = ""
--> }
--> Args:
-->
--> Arg configSpec:
--> (vim.vcha.FailoverClusterConfigurator.VchaClusterConfigSpec) {
--> passiveIp = "#.#.#.#",
--> witnessIp = "#.#.#.#"
--> }
[YYYY:MM:DDTHH:MM:SS] info vpxd[7FC49E8D1700] [Originator@6876 sub=vpxLro opID=dam-auto-generated: GenerationNumbersMonitor:dr-407:01-4b] [VpxLRO] -- BEGIN lro-3739 -- ResourceModel -- cis.data.provider.ResourceModel.query -- ########-####-####-####-###########(########-####-####-####-###########)

 

• When testing network communication between Active and Passive nodes the connections fail or receive no response.
  • Connect to either the Active or Passive node via SSH.
  • curl -v telnet://<ip_of_peer_node>:22
  • ping <ip_of_peer_node>

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

• vCenter Server Appliance 6.5
• vCenter Server Appliance 7.0
• vCenter Server Appliance 8.0

Cause

• If the correct order is not followed to start the clone of Active node to create Passive and Witness VM, resulting in a situation where SSH keys are not present on the passive/witness VMs at the time of the connection attempt.
• There are left-over passive/witness nodes from previous configurations of VCHA, or previously failed attempts to configure VCHA
• If the there are improper network configuration. 
• After destroying VCHA, VCHA nic from the previous VCHA configuration is still present on the active node.

Resolution

• When using the VCHA Advanced option in vCenter server appliance, strictly follow the order to start the clone of Active node to create Passive and Witness VMs only after wizard reaches a certain point and asks user to start the clone operation.
• Ensure there are no old passive/witness VMs from previous VCHA configurations or failed configuration attempts.
• Ensure the configuration, specifically networking, is correct for the environment.