VMware vCenter Server displays the error: Failed to connect to host

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

Symptoms:

vCenter Server cannot connect to a host when performing operations such as:
- Storage vMotion
- Cold migration
- Cloning a virtual machine
- Deploying from template
In the vpxa logs, you see entries similar to:

info 'Libs' opID=E3E07CA2-0000CC13-94] Cannot connect to server x.x.x.x: Connection timed outinfo 'Libs' opID=E3E07CA2-0000CC13-94] CnxAuthdConnect: Returning false because CnxAuthdConnectTCP failed
info 'Libs' opID=E3E07CA2-0000CC13-94] CnxConnectAuthd: Returning false because CnxAuthdConnect failed
info 'Libs' opID=E3E07CA2-0000CC13-94] Cnx_Connect: Returning false because CnxConnectAuthd failed
[2012-03-08 20:44:00.670 19F3FB90 info 'Libs' opID=E3E07CA2-0000CC13-94] Cnx_Connect: Error message: Failed to connect to server x.x.x.x:902
warning 'Libs' opID=E3E07CA2-0000CC13-94] [NFC ERROR] NfcNewAuthdConnectionEx: Failed to connect to peer (numRetries=0). Error: Failed to connect to server x.x.x.x:902
In the vpxd logs, you see an entry similar to:

[YYYY-MM-DDTHH:MM:SS] 'App' 4444 error [VpxdInvtHost] IP address change for x.x.x.x to x.x.x.x not handled, SSL certificate verification is not enabled.
You see entries similar to:
- Failed To Connect to host
- [YYYY-MM-DDTHH:MM:SS] 'Libs' 3902384 warning [NFC ERROR] NfcNewAuthdConnectionEx: Failed to connect to peer (numRetries=2). Error: Cannot connect to host x.x.x.x: Connection timed out
vCenter Server shows virtual machines as being disconnected then connected.

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware ESXi 4.0.x Installable
VMware ESXi 4.1.x Embedded
VMware ESX 4.0.x
VMware vCenter Server 4.1.x
VMware ESX 4.1.x
VMware vCenter Server 5.1.x
VMware vSphere ESXi 5.0
VMware vSphere ESXi 5.5
VMware vCenter Server 5.0.x
VMware vCenter Server 5.5.x
VMware ESXi 4.0.x Embedded
VMware VirtualCenter 2.5.x
VMware ESXi 4.1.x Installable
VMware vSphere ESXi 5.1
VMware vCenter Server 4.0.x

Cause

This issue occurs if the IP address of an ESX host is changed while being managed by vCenter Server. If the vCenter Server IP has been changed, see VMware vCenter Server IP address change causes VMware ESX hosts to disconnect (337654).

To verify the cause of the issue:

Log in as root to the ESX host using an SSH client.
In the /etc/opt/vmware/vpxa/vpxa.cfg file, locate the Host IP address. In ESXi 5.x, the configuration file is located at /etc/vmware/vpxa/vpxa.cfg file. You see entries similar to:
<vpxa>
<hostIp>x.x.x.x</hostIp>
</vpxa>

Resolution

To resolve this issue:

Note: If the IP address is incorrect, perform these steps in order. Do not skip a step:

Disconnect and reconnect the host:
1. Right-click the ESX host in vCenter Server and click Disconnect.
2. Reconnect the ESX host in vCenter Server by the Management IP address, not the hostname.
Restart the VirtualCenter agent on the ESX host:
1. Run this command to stop the vpxa service:
  
  # service vmware-vpxa stop
  
  Run this command in the Tech Support mode to stop the vpxa service in ESXi 5.x:
  
  # /etc/init.d/vpxa stop
2. Open the /etc/opt/vmware/vpxa/vpxa.cfg file using the text editor and correct the IP address of the ESX host.
3. Run this command to start the vpxa service:
  
  # service vmware-vpxa start
  
  Run this command in the Tech Support mode to start the vpxa service in ESXi 5.x:
  
  # /etc/init.d/vpxa start
  
  Note: For VMware ESXi you may need to restart all of the management agents. For more information, see Restarting the Management agents on an ESXi or ESX host (320280) .
Rebuild the VirtualCenter agent configuration file on the ESX host:

Caution: This step removes all host and virtual machine historical statistics counters. If the VMware ESX host is running virtual machines that started up in an EVC enabled environment, you may not be able to add the host back without shutting down the virtual machines first.
1. Right-click the ESX host in vCenter Server and click Disconnect.
2. Remove the disconnected ESX host from vCenter Server.
  
  Note: It has been reported that in some instances it may be necessary to disable SSL certificates in the vCenter Server > Administration > vCenter Server Settings > SSL Settings for the update to vpxd.cfg to work . The SSL certificates can be re-enabled once the ESX host has been re-added back to the vCenter Server inventory .
3. Run this command to take a backup the vpxa.cfg file:
  
  Note: In ESXi 5.x, the vpxa.cfg configuration file is located at /etc/vmware/vpxa/
  
  # mv /etc/vmware/vpxa/vpxa.cfg /etc/vmware/vpxa/vpxa.oldcfg
4. Add the disconnected ESX host to the vCenter Server inventory.
5. Run this command to examine the contents of /etc/opt/vmware/vpxa/vpxa.cfg and confirm that the host IP address is correct:
  
  # cat /etc/opt/vmware/vpxa/vpxa.cfg
  
  In ESXi 5.x, run the command:
  
  # cat /etc/vmware/vpxa/vpxa.cfg

Additional Information

If the above steps do not resolve the issue, attempt to uninstall and install the agents on the VMware ESX host. For more information, see Verifying and reinstalling the correct version of the vCenter Server agents (308505).

Note: If the ESX host is behind a NAT and the vCenter Server is on a different network, the above workaround may not address these issues or may impact other features such as HA which requires ESX hosts to communicate with each other through the actual IP and not NAT. For more information, see Using NAT between the vCenter Server system and ESXi/ESX hosts (344283). This issue may also occur if NAT is turned on for the internal networks.

VMware vCenter Server IP address change causes VMware ESX hosts to disconnect
Verifying and reinstalling the correct version of the VMware vCenter Server agents
Using NAT between the vCenter Server system and ESXi/ESX hosts