Adding host to cluster fails at 'ESXi host Remediate stage' with an error "Applying personality to cluster domain-### failed with error: Health Check for domain-c### failed"

Products

• VMware Cloud Foundation
• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• After host commission, failed to add new host to cluster at remediate task with below error

Remediate ESXi Host(s) to be Compliant with Cluster's Image
Applying personality to cluster domain-c### failed with error: com.vmware.vapi.std.errors.Error: Error (com.vmware.vapi.std.errors.error) => { messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => { id = com.vmware.vcintegrity.lifecycle.TaskError.HealthCheckFailed, defaultMessage = Health Check for <name>
Health Check for "'name' failed }], data = <null>, errorType = ERROR }

Message: Applying personality to cluster domain-### failed with error.com.vmware.vapi,std.errors.Error: Error (com.vmware.vapi.std.errors.error) =>{ messages = [LodalizableMessage(com.vmware.vapi.std.localizable_message) => { id =com.vmware.vcintegrity.lifecycle.TaskError.HealthCheckFailed,defaultMessage = Health Check for 'name' failed, args = [name], params = <null>, localized = Health Check for '<name> failed }], data = <null>, errorType = ERROR }
Remediation Message:
Reterence Token: 59A77D

• In VCF Manager - /var/log/vmware/vcf/domainmanager/domainmanager.log
YYYY-MM-DDTHH:MM:SS ERROR [vcf_dm,697254516ed0c36e57996c9a4050f7c7,8c94] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-24]  [59A77D] VLCM_REMEDIATE_PERSONALITY_FAILED Applying personality to cluster domain-c### failed with error: com.vmware.vapi.std.errors.Error: Error (com.vmware.vapi.std.errors.error) => {
    messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
    id = com.vmware.vcIntegrity.lifecycle.TaskError.HealthCheckFailed,
    defaultMessage = Health Check for 'domain-c###' failed,
    args = [domain-c###],
    params = <null>,
    localized = Health Check for 'host_fqdn' failed
}],
    data = <null>,
    errorType = ERROR

• In  VC - /var/log/vmware/vmware-updatemgr/vum-server.log
YYYY-MM-DDTHH:MM:SS error vmware-vum-server[3462450] [Originator@6876 sub=RemediateClusterTask] [Task, 424] Task:com.vmware.vcIntegrity.lifecycle.RemediateClusterTask ID:<id>. Task Failed. Error
-->    com.vmware.vapi.std.errors.error
--> Messages:
-->    com.vmware.vcIntegrity.lifecycle.TaskError.HealthCheckFailed<Health Check for 'domain-###' failed>
-->
YYYY-MM-DDTHH:MM:SS warning vmware-vum-server[3608627] [Originator@6876 sub=TaskStatsCollector] [taskStatsCollector 190] Task type or creation time not present
YYYY-MM-DDTHH:MM:SS info vmware-vum-server[3462450] [Originator@6876 sub=PM.AsyncTask.RemediateClusterTask{80}] [vciTaskBase 1493] SerializeToVimFault fault:
--> (vmodl.fault.SystemError) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = (vmodl.LocalizableMessage) [
-->       (vmodl.LocalizableMessage) {
-->          key = "com.vmware.vcIntegrity.lifecycle.TaskError.HealthCheckFailed",
-->          arg = (vmodl.KeyAnyValue) [
-->             (vmodl.KeyAnyValue) {
-->                key = "1",
-->                value = "domain-c###"
-->             }
-->          ],
-->          message = <unset>
-->       }
-->    ],
-->    reason = "vLCM Task failed, see Error Stack for details."
-->    msg = "{
-->     "data": null,
-->     "error_type": "ERROR",
-->     "messages": [
-->         {
-->             "args": [
-->                 "domain-c###"
-->             ],
-->             "default_message": "Health Check for 'name' failed",
-->             "id": "com.vmware.vcIntegrity.lifecycle.TaskError.HealthCheckFailed"

Environment

• VCF 9.x
• vCenter 9.x
• ESX 9.x

Cause

Remediation fails due to an alert on vSAN health, reported for the cluster --- 'SCSI controller is VMware-certified'

Resolution

1. Validate if the SCSI controller is certified in Broadcom compatibility Guide using following link:
   
   vSAN I/O Controller
   
   
2. If it is listed in the compatibility guide, alert can be silenced.
   
   (Cluster > Monitor > vSAN > Skyline Health > under alert 'SCSI Controller is VMware Certified > Click on 'Silence Alert')


3. Return to the VCF Manager UI and restart the failed task by clicking the ‘Restart Task‘ button