Configuring vSphere HA on a cluster fails due to unknown host status when checking image compliance

Products

• VMware vSphere ESXi

Issue/Introduction

• HA cannot be enabled on an ESXi host in an image-based cluster.
• The "Remediate HA" task fails with the error "

A general system error occurred: Installing HA components failed on the cluster: domain-c#"

• The "Configuring vSphere HA on the cluster" task fails with the error:

Cannot complete the configuration of the vSphere HA agent on the host. "Applying HA VIBs on the cluster encountered a failure". Failed installing HA components on the host: host-##

In vCenter /var/log/vmware/vmware-updatemgr/vum-server/vum-server.log:

YYYY-MM-DDTHH:MM:13.822Z info vmware-vum-server[933528] [Originator@6876 sub=PM.AsyncTask.ClusterApplyHATask{2253}] [vciTaskBase 1496] SerializeToVimFault fault:
--> (vmodl.fault.SystemError) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = (vmodl.LocalizableMessage) [
-->       (vmodl.LocalizableMessage) {
-->          key = "com.vmware.vcIntegrity.lifecycle.ClusterApplyHATask.ApplyHATaskFailed",
-->          arg = (vmodl.KeyAnyValue) [
-->             (vmodl.KeyAnyValue) {
-->                key = "1",
-->                value = "domain-####"
-->             }
-->          ],
-->          message = <unset>
-->       }
-->    ],
-->    reason = "vLCM Task failed, see Error Stack for details."
-->    msg = "{
-->     "data": null,
-->     "error_type": "ERROR",
-->     "messages": [
-->         {
-->             "args": [
-->                 "domain-####"
-->             ],
-->             "default_message": "Installing HA components failed on the cluster: domain-####",
-->             "id": "com.vmware.vcIntegrity.lifecycle.ClusterApplyHATask.ApplyHATaskFailed"
-->         }
-->     ]
--> }"
--> }

• The "Reconfigure for VMware HA" option is greyed out for the affected ESXi host.
• When attempting to check compliance for the host against the image, this fails with the error:

Hosts status is unknown "a failure occurred when starting the host compliance check operation on host "hostname.domain.local' : Error: com.vmware.std.errors.not_allowed_in_current_state Messages: com.vmware.esx.task.contention<Another task is in progress. Please retry later.>

• Check the settingsd is running in the ESXi using /etc/init.d/settingsd status

/etc/init.d/settingsd status
settingsd is not running

Environment

VMware vCenter Server 8.x
VMware vSphere ESXi 8.x

Cause

The FDM VIB was not installed on the ESXi host. This was due to the compliance check failing. If the compliance check fails, then the host can't be remediated against the image.

Resolution

1. Restart the settingsd management daemon on the affected ESXi host:

/etc/init.d/settingsd restart

Note: If the settingsd service appears as “not running” after a restart, manually terminate the existing process and then start the service again.

To identify the process ID (PID)::  ps | grep settingsd

[root@<hostname> ~] ps | grep settingsd
2102050    2102050    settingsd

To terminate the process::  kill -s SIGABRT < PID> 

2. Re-run the Check Compliance task in Image compliance (Navigate to the cluster, then select the Updates tab)

3. Once the compliance check completes and the host shows "Host is out of compliance with the image", remediate the host against the image.

4. Disable HA on the cluster and re-enable HA, see Disabling and enabling VMware High Availability (HA)

5. Finally, confirm if HA is enabled on the ESXi host (Host -> Summary tab -> Configuration -> vSphere HA state)