Tasks stuck at 0% for a specific ESXi host on vCenter Server UI
search cancel

Tasks stuck at 0% for a specific ESXi host on vCenter Server UI

book

Article ID: 413660

calendar_today

Updated On:

Products

VMware vSphere ESXi VMware vCenter Server

Issue/Introduction

Tasks initiated from the vCenter Server UI for a specific ESXi host remain stuck at 0% and do not progress. This issue indicates a communication problem between the vCenter Server and the affected ESXi host, preventing the vCenter Server from receiving updates or successfully issuing commands to the host.

  • Tasks (e.g., vMotion, Storage vMotion, configuration changes) initiated from vCenter Server UI for a particular ESXi host are stuck at 0%.
  • The vCenter Server may report "No heartbeats received from host" in its logs.
  • The management network address of the ESXi host was recently changed.


esxcli system syslog tail -f /var/run/log/vmkernel.log (on the ESXi host):
Observe entries indicating a recent change in the management IP address:

2025-10-01T07:33:49.229Z In(182) vmkernel: cpu19:2101084)Config: 833: "HostIPAddr" = "#.#.#.x", Old value: "#.#.#.y" (Status: 0x0)
2025-10-01T07:33:49.232Z In(182) vmkernel: cpu19:2101084)Config: 833: "ManagementAddr" = "#.#.#.x", Old value: "#.#.#.y" (Status: 0x0)

(Note: #.#.#.# represents the actual IP addresses)

/var/log/vmware/vpxd/vpxd.log (on the vCenter Server Appliance):
Observe entries indicating missed heartbeats from the ESXi host:

2025-10-01T15:36:11.275+08:00 info vpxd[06000] [Originator@6876 sub=HostCnx opID=CheckforMissingHeartbeats-5db39dc4] [VpxdHostCnx] No heartbeats received from host; cnx: 5######5-3##c-8##3-f##0-a###########1, h: host-4####1, time since last heartbeat: 66835ms

(Note: Hexadecimal strings represent internal IDs and will vary)

 

Environment

  • VMware ESXi (all versions)
  • VMware vCenter Server (all versions)

Cause

The primary cause of this issue is a communication breakdown between the vCenter Server and the ESXi host, specifically regarding the exchange of heartbeats over UDP port 902. This can be triggered by a recent change in the ESXi host's management network IP address which was not fully reconciled or by network devices (e.g., firewalls) blocking this essential heartbeat communication.

Resolution

To troubleshoot and resolve tasks stuck at 0% due to missed heartbeats, follow these steps:

  1. Verify Heartbeat Packet Transmission from ESXi Host:
    Use the pktcap-uw command on the affected ESXi host to check if it's actively sending heartbeat packets to the vCenter Server. Replace vmnicx with the actual uplink associated with the host's management network.

    esxi# pktcap-uw --uplink vmnicx --capture UplinkSndKernel --udpport 902 -o - | tcpdump-uw -enr -

    This command captures outgoing UDP 902 packets from the ESXi host. If no packets are displayed, the ESXi host is not sending heartbeats, or they are not exiting the specified vmnic. If packets are displayed, it confirms the ESXi host is attempting to send heartbeats.

  2. Verify Heartbeat Packet Reception on vCenter Server:
    Similarly, perform a packet capture on the vCenter Server to see if it is receiving the heartbeat packets from the ESXi host.

    • For vCenter Server Appliance (VCSA):

      vcsa# tcpdump -i <interface_name> host <ESXi_IP_Address> and udp port 902

      (Replace <interface_name> with the appropriate network interface, e.g., eth0, and <ESXi_IP_Address> with the IP of the problematic ESXi host).

    • For Windows-based vCenter Server: Use a tool like Wireshark or netsh trace to capture traffic on UDP port 902 coming from the ESXi host.

  3. Check Network Path and Firewall Configuration:

    • Identify Network Devices: Map out all network devices (switches, routers, firewalls) between the ESXi host and the vCenter Server.
    • Firewall Involvement: If firewalls are present in the communication path, specifically investigate if UDP port 902 is being blocked. Engage your network or firewall administration team to:
      • Confirm that UDP port 902 is open bidirectionally between the ESXi host's management IP and the vCenter Server's IP address.
      • Check for any Network Address Translation (NAT) or Access Control List (ACL) rules that might be inadvertently interfering with UDP 902 traffic.

  4. Reconnect the ESXi Host to vCenter (if necessary):
    If the IP address was changed and communication issues persist after verifying network connectivity, you may need to disconnect and then reconnect the ESXi host to vCenter.

    • Right-click the ESXi host in vCenter Server and select Connection > Disconnect.
    • Once disconnected, right-click the ESXi host again and select Connection > Connect.
    • If reconnection fails, you might need to remove the host from inventory and re-add it (ensure VMs are shut down or migrated off first, if possible).

  5. Restart Management Agents (as a last resort on the ESXi host):
    If basic network connectivity appears fine but heartbeats are still an issue, restarting the management agents on the ESXi host can sometimes resolve transient communication glitches.

    • Connect to the ESXi host via SSH or Console.
    • Run:
      bash /etc/init.d/hostd restart /etc/init.d/vpxa restart
    • Wait a few minutes for the agents to restart and for the host to reconnect to vCenter.

By systematically following these steps, you should be able to diagnose and resolve the communication issue causing tasks to be stuck at 0% for the specific ESXi host.

Additional Information

Powered by Wolken Software