HA configuration task is stuck for long time or fails with Configuration error after an ESXi host update in the cluster
search cancel

HA configuration task is stuck for long time or fails with Configuration error after an ESXi host update in the cluster

book

Article ID: 369999

calendar_today

Updated On:

Products

VMware vCenter Server 7.0 VMware vSphere ESXi VMware vSphere ESXi 7.0

Issue/Introduction

1. After a recent ESXi host update the HA configuration is stuck for long time

2. HA configuration fails with "Configuration error"

3. In the vCenter vpxd log located at /var/log/vmware/vpxd/vpxd.log it shows below similar entries

info vpxd[40638] [Originator@6876 sub=MoHost opID=lro-623-62130020] VC state for host host-848440 (uninitialized -> retry), FDM state (UNKNOWN_FDM_HSTATE -> UNKNOWN_FDM_HSTATE), src of state (null -> null)
info vpxd[40231] [Originator@6876 sub=MoHost opID=lro-616-7e03148e] VC state for host host-826061 (uninitialized -> retry), FDM state (UNKNOWN_FDM_HSTATE -> UNKNOWN_FDM_HSTATE), src of state (null -> null)
info vpxd[39820] [Originator@6876 sub=MoHost opID=lro-617-7c3ab743] VC state for host host-122893 (uninitialized -> retry), FDM state (UNKNOWN_FDM_HSTATE -> UNKNOWN_FDM_HSTATE), src of state (null -> null)
info vpxd[40628] [Originator@6876 sub=MoHost opID=lro-626-1d1d2032] VC state for host host-1036579 (uninitialized -> retry), FDM state (UNKNOWN_FDM_HSTATE -> UNKNOWN_FDM_HSTATE), src of state (null -> null)
info vpxd[40579] [Originator@6876 sub=MoHost opID=lro-621-15ffbc01] VC state for host host-397994 (uninitialized -> retry), FDM state (UNKNOWN_FDM_HSTATE -> UNKNOWN_FDM_HSTATE), src of state (null -> null)
info vpxd[39915] [Originator@6876 sub=MoHost opID=lro-634-98eaf24] VC state for host host-145375 (uninitialized -> retry), FDM state (UNKNOWN_FDM_HSTATE -> UNKNOWN_FDM_HSTATE), src of state (null -> null)
info vpxd[40759] [Originator@6876 sub=MoHost opID=FdmMonitor-domain-c981493-12f30c61] VC state for host host-1010562 (uninitialized -> uninitialized), FDM state (Master -> UNKNOWN_FDM_HSTATE), src of state (null -> null)
info vpxd[40638] [Originator@6876 sub=MoHost opID=lro-790-ce81c45] VC state for host host-1081598 (retry -> uninitialized), FDM state (UNKNOWN_FDM_HSTATE -> UNKNOWN_FDM_HSTATE), src of state (null -> null)
info vpxd[40638] [Originator@6876 sub=MoHost opID=lro-790-ce81c45] VC state for host host-1081592 (retry -> uninitialized), FDM state (UNKNOWN_FDM_HSTATE -> UNKNOWN_FDM_HSTATE), src of state (null -> null)
info vpxd[40638] [Originator@6876 sub=MoHost opID=lro-790-ce81c45] VC state for host host-1081595 (retry -> uninitialized), FDM state (UNKNOWN_FDM_HSTATE -> UNKNOWN_FDM_HSTATE), src of state (null -> null)
info vpxd[40440] [Originator@6876 sub=MoHost opID=lro-838-5391b34f] VC state for host host-834347 (retry -> uninitialized), FDM state (UNKNOWN_FDM_HSTATE -> UNKNOWN_FDM_HSTATE), src of state (null -> null)
info vpxd[40440] [Originator@6876 sub=MoHost opID=lro-838-5391b34f] VC state for host host-1068172 (retry -> uninitialized), FDM state (UNKNOWN_FDM_HSTATE -> UNKNOWN_FDM_HSTATE), src of state (null -> null)
info vpxd[40440] [Originator@6876 sub=MoHost opID=lro-838-5391b34f] VC state for host host-834350 (retry -> uninitialized), FDM state (UNKNOWN_FDM_HSTATE -> UNKNOWN_FDM_HSTATE), src of state (null -> null)
info vpxd[40440] [Originator@6876 sub=MoHost opID=lro-838-5391b34f] VC state for host host-834344 (retry -> uninitialized), FDM state (UNKNOWN_FDM_HSTATE -> UNKNOWN_FDM_HSTATE), src of state (null -> null)
info vpxd[39814] [Originator@6876 sub=MoHost opID=lro-843-49134a39] VC state for host host-380126 (retry -> uninitialized), FDM state (UNKNOWN_FDM_HSTATE -> UNKNOWN_FDM_HSTATE), src of state (null -> null)
info vpxd[39814] [Originator@6876 sub=MoHost opID=lro-843-49134a39] VC state for host host-380123 (retry -> uninitialized), FDM state (UNKNOWN_FDM_HSTATE -> UNKNOWN_FDM_HSTATE), src of state (null -> null)
info vpxd[40440] [Originator@6876 sub=MoHost opID=lro-838-5391b34f] VC state for host host-1078304 (uninitialized -> retry), FDM state (UNKNOWN_FDM_HSTATE -> UNKNOWN_FDM_HSTATE), src of state (null -> null)
info vpxd[40440] [Originator@6876 sub=MoHost opID=lro-838-5391b34f] VC state for host host-805681 (uninitialized -> retry), FDM state (UNKNOWN_FDM_HSTATE -> UNKNOWN_FDM_HSTATE), src of state (null -> null)
info vpxd[40440] [Originator@6876 sub=MoHost opID=lro-838-5391b34f] VC state for host host-834347 (uninitialized -> retry), FDM state (UNKNOWN_FDM_HSTATE -> UNKNOWN_FDM_HSTATE), src of state (null -> null)
info vpxd[40440] [Originator@6876 sub=MoHost opID=lro-838-5391b34f] VC state for host host-1068172 (uninitialized -> retry), FDM state (UNKNOWN_FDM_HSTATE -> UNKNOWN_FDM_HSTATE), src of state (null -> null)
info vpxd[40440] [Originator@6876 sub=MoHost opID=lro-838-5391b34f] VC state for host host-834350 (uninitialized -> retry), FDM state (UNKNOWN_FDM_HSTATE -> UNKNOWN_FDM_HSTATE), src of state (null -> null)
info pxd[40440] [Originator@6876 sub=MoHost opID=lro-838-5391b34f] VC state for host host-834344 (uninitialized -> retry), FDM state (UNKNOWN_FDM_HSTATE -> UNKNOWN_FDM_HSTATE), src of state (null -> null)
info vpxd[39505] [Originator@6876 sub=MoHost opID=lro-849-6b585da3] VC state for host host-145375 (retry -> uninitialized), FDM state (UNKNOWN_FDM_HSTATE -> UNKNOWN_FDM_HSTATE), src of state (null -> null)

4. In the vpxd-profiler.log located at /var/log/vmware/vpxd will show similar entries as below

warning vpxd[08461] [Originator@6876 sub=DAS opID=lro-1145282-333db184] Failed to get task ########-####-####-####-########d926:com.vmware.esx.settings.clusters.software.ha_internal status due to  error: com.vmware.vapi.std.errors.unauthenticated.Retrying it again with new session.
warning vpxd[06837] [Originator@6876 sub=DAS opID=lro-1168331-684c3cd0] Failed to get task ########-####-####-####-########34b1:com.vmware.esx.settings.clusters.software.ha_internal status due to  error: com.vmware.vapi.std.errors.unauthenticated.Retrying it again with new session.
warning vpxd[08678] [Originator@6876 sub=DAS opID=lro-1191620-10ab0b74] Failed to get task ########-####-####-####-########1e01:com.vmware.esx.settings.clusters.software.ha_internal status due to  error: com.vmware.vapi.std.errors.unauthenticated.Retrying it again with new session.
warning vpxd[08607] [Originator@6876 sub=DAS opID=lro-1216414-24723b00] Failed to get task ########-####-####-####-########71e9:com.vmware.esx.settings.clusters.software.ha_internal status due to  error: com.vmware.vapi.std.errors.unauthenticated.Retrying it again with new session.
warning vpxd[08678] [Originator@6876 sub=DAS opID=lro-1191620-10ab0b74] Failed to get task ########-####-####-####-########1e01:com.vmware.esx.settings.clusters.software.ha_internal status due to  error: com.vmware.vapi.std.errors.unauthenticated.Retrying it again with new session.
warning vpxd[08607] [Originator@6876 sub=DAS opID=lro-1216414-24723b00] Failed to get task ########-####-####-####-########71e9:com.vmware.esx.settings.clusters.software.ha_internal status due to  error: com.vmware.vapi.std.errors.unauthenticated.Retrying it again with new session

 

You will also see task waitForUpdates in vpxd-profiler.log

ThreadState/ThreadId/48605/State/RPC::vmodl.query.PropertyCollector:propertyCollector::<hostname>::vmodl.query.PropertyCollector.waitForUpdates



5. In the ESXi host, hostd.log located at  /var/run/log will show similar entries as below

info hostd[2102049] [Originator@6876 sub=SysCommandPosix opID=lxd8ztio-95309-auto-21ji-h5:700120##-##-##-##-##02 user=vpxuser] ForkExec(/sbin/chkconfig) 2259639
info hostd[2102049] [Originator@6876 sub=Vimsvc.TaskManager opID=lxd8ztio-95309-auto-21ji-h5:700120##-##-##-##-##02 user=vpxuser] Task Completed : haTask-ha-host-vim.host.ServiceSystem.updatePolicy-476801 Status success
info hostd[2102035] [Originator@6876 sub=Vimsvc.TaskManager opID=lxd8ztio-95309-auto-21ji-h5:700120##-##-##-##-##07 user=vpxuser] Task Created : haTask-ha-host-vim.host.ServiceSystem.stop-476802
error hostd[2102035] [Originator@6876 sub=Libs opID=lxd8ztio-95309-auto-21ji-h5:700120##-##-##-##-##07 user=vpxuser] ServiceAgent: Command(/etc/init.d/vmware-fdm ++group=hostd-tmp status) exited with status 256
info hostd[2102035] [Originator@6876 sub=Vimsvc.TaskManager opID=lxd8ztio-95309-auto-21ji-h5:700120##-##-##-##-##07 user=vpxuser] Task Completed : haTask-ha-host-vim.host.ServiceSystem.stop-476802 Status success



 

 

Environment

VMware vCenter Server 7.0.x
VMware vSphere ESXi 7.0.x

Cause

After an ESXi update, a race condition might cause the image compliance check to take longer than usual and lead to issues such as time out of the vSphere HA reconfiguration after the update.

Resolution

This issue is resolved in VMware vCenter 7.0 U3q version. To download go to VCF Products download page.

To workaround this issue Enter/Exit  maintenance mode manually and configure HA.

NOTE: You should apply the workaround before updating the vCenter server to fixed version.

 

 

Additional Information

Refer to VMware vCenter 7.0U3q release notes

PR 3279536: vSphere High Availability (HA) reconfiguration might time out due to a delay in image compliance check tasks.
After an ESXi update, a race condition might cause the image compliance check to take longer than usual and lead to issues such as time out of the vSphere HA reconfiguration after the update.

This issue is resolved in this release.

Powered by Wolken Software