vSphere HA initialization task queued behind stale tasks

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

The article provides steps to address vSphere HA configuration issue by analyzing the logs and reviewing stale/pending tasks.

• vSphere HA restart task gets stuck for one or two ESX hosts.
• Option to place host in maintenance mode or disconnect/reconnect options are greyed out.
• FDM agent communication issue between affect ESX host and primary host
  
  In /var/run/log/fdm.log, we read the following:

YYYY-MM-DDTHH:MIN:SEC In(166) Fdm[#####]: [Originator@6876 sub=Cluster opID=clusterManager.cpp:#####-####] Adding local datastore /vmfs/volumes/####-####
YYYY-MM-DDTHH:MIN:SEC In(166) Fdm[#####]: [Originator@6876 sub=Cluster opID=clusterManager.cpp:#####-####] Adding Datastore : path=/vmfs/volumes/####-#### mountHost=host-#### type=1 accessible=true
YYYY-MM-DDTHH:MIN:SEC In(166) Fdm[#####]: [Originator@6876 sub=Election opID=clusterElection.cpp:#####-####] Not running election
YYYY-MM-DDTHH:MIN:SEC In(166) Fdm[#####]: [Originator@6876 sub=Cluster opID=clusterManager.cpp:#####-####] Adding Datastore : path=/vmfs/volumes/####-#### mountHost=host-#### type=1 accessible=true

In /var/run/log/fdm.log of primary host:

YYYY-MM-DDTHH:MIN:SEC Db(167) Fdm[#####]: [Originator@6876 sub=Cluster opID=#####-####] Remote host FdmVersion object not found; remoteHost: host-####
YYYY-MM-DDTHH:MIN:SEC Db(167) Fdm[#####]: [Originator@6876 sub=Cluster opID=#####-####] Remote host FdmVersion object not found; remoteHost: host-####



Cause

• When multiple ESX hosts are disconnected or unresponsvie due to storage or network outage occurred in datacenter, the vSphere HA shall be reinitialized once ESX hosts are available. If the said task is queued up in the list of tasks that has been stalled or unattended, the vSphere HA configuration shall not be complete and this will lead to such behavior.
• In the below image, we read, configure vSphere HA being queued up behind Disconnect Host. The task is listed in vCenter UI task list and both tasks belong to the same ESX host.
  
  

Resolution

• To clear all the tasks that is queued and stale tasks, restart VPXD service and restart vSphere HA agent for affected ESX host/s.

Note: Review the list of task which is in active state. Tasks such as scheduled backup, VDI deployment or monitoring alerts shall be impacted until the service is active.

1. Open SSH connection to vCenter server.
2. Login as root user
3. Restart vpxd service: system-control --restart vmware-vpxd
4. Confirm active status of vpxd service: system-control --status vmware-vpxd
5. Restart vSphere HA Agent on an affected ESX host by following the steps mentioned in Broadcom article vSphere HA Agent Alert: Cannot Reach Management Network Addresses