ESXi host disconnects intermittently from vCenter Server
search cancel

ESXi host disconnects intermittently from vCenter Server

book

Article ID: 318647

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:
  • ESX/ESXi hosts disconnect frequently from vCenter Server.
  • vCenter Server shows ESX/ESXi host(s) as not responding
  • vCenter Server is not randomly receiving ESX/ESXi heartbeats
  • When verbose logging is enabled, in the C:\ProgramData\VMware\VMware VirtualCenter\Logs\vpxd.log (/var/log/vmware/vpxd/ in vCenter Server Appliance) file, you see entries similar to:

    [<YYYY-MM-DD>T<time> verbose 'App'] [VpxdIntHost] Missed 2 heartbeats for host esx03.it.abc.local

    </time>
Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
 


Environment

VMware vCenter Server 6.7.x
VMware vCenter Server Appliance 5.5.x
VMware vCenter Server Appliance 6.0.x
VMware vCenter Server 7.0.x
VMware vCenter Server Appliance 6.5.x
VMware vCenter Server 6.0.x
VMware vCenter Server 5.5.x
VMware vCenter Server Appliance 6.7.x
VMware vCenter Server 6.5.x

Cause

This issue occurs when the UDP heartbeat message sent by ESX/ESXi host is not received by vCenter Server. if vCenter Server does not receive the UDP heartbeat message, it treats the host as not responding. ESX/ESXi host send heartbeats every 10 seconds and vCenter Server has a window of 60 seconds to receive the heartbeats. This behavior can be an indication of a congested network between the ESX/ESXi host and vCenter Server.

Note: If the host disconnects every 60 seconds there is likely a firewall blocking UDP 902 heartbeats from ESXi host to vCenter. 

Resolution

Issue has to be investigated via packet capture to eliminate any packet drops or handshake issues.
In the meanwhile, you can follow the workaround mentioned in this KB.


Workaround:
To work around this issue, increase the timeout limit in vCenter Server by editing or creating the Advanced Setting config.vpxd.heartbeat.notRespondingTimeout.

Note: Increasing the timeout is a short-term solution until any network issues can be resolved.
 
To increase the timeout limit:
 
vSphere Client (C#):
  1. Open the vSphere Client.
  2. Connect to vCenter Server.
  3. Select Administration > vCenter Server Settings to display the vCenter Server Settings dialog box.
  4. If the vCenter Server system is part of a connected group, select the server you want to configure from the Current vCenter Server drop-down.
  5. In the settings list, select Advanced Settings.
  6. In the Key field, type:

    config.vpxd.heartbeat.notRespondingTimeout
     
  7. In the Value field, type:

    120
     
  8. Click Add.
  9. Click OK.
  10. Restart the vCenter Server service.
vSphere Web Client (Flash/Flex) & vSphere Client (HTML5):
  1. Open the vSphere Web Client or vSphere Client in a web browser and log in.
  2. Select the vCenter object from the inventory under Hosts and Clusters.
  3. Select the Manage or Configure tab.
  4. Select SettingsAdvanced Settings.
  5. Click Edit.
  6. In the Key field, type:

    config.vpxd.heartbeat.notRespondingTimeout
     
  7. In the Value field, type:

    120
     
  8. Click Add.
  9. Click OK.
  10. Restart the vCenter Server service.
vCenter Server Appliance 5.5: service vmware-vpxd restart
vCenter Server Appliance 6.x and later: service-control --stop vmware-vpxd; service-control --start vmware-vpxd


vCenter 6.5 (windows)

1. Navigate to:
C:\ProgramData\VMware\vCenterServer\cfg\vmware-vpx

2. Add the following entry in vpxd.cfg:

<notRespondingTimeout>120</notRespondingTimeout>

Above entry was added between following (there can be one more entry for port, put the heartbeat entry below it):
<heartbeat>
</heartbeat>

3. Restart the vpxd service:

Open elevated command prompt and navigate to the directory "C:\Program Files\VMware\vCenter Server\bin"

Run the following:

service-control --stop vpxd
service-control --start vpxd

4. Log out and log into the web client and the advanced setting entry will be added: "config.vpxd.heartbeat.notRespondingTimeout"

For additional details on restarting vCenter Server services see How to Stop, Start or Restart vCenter Server 6.x Services (2109881).


Additional Information