SDDC Manager Add host to cluster task fails at remediate ESXi Host(s) to be Compliant with Cluster's Image with error message 'VLCM_REMEDIATE_PERSONALITY_FAILED'

Products

• VMware SDDC Manager
• VMware Cloud Foundation

Issue/Introduction

When adding a host to an existing cluster fails at remediate ESXi Host(s) to be Compliant with Cluster's Image with error message 'VLCM_REMEDIATE_PERSONALITY_FAILED'.

domainmanager.log reports health check failures:

/var/log/vmware/vcf/domainmanager/domainmanager.log

YYYY-MM-DDTHH:MM:SS.145+0000 ERROR [vcf_dm,########-####-####-####-############,07ae] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-28]  [N#####] VLCM_REMEDIATE_PERSONALITY_FAILED Applying personality to cluster domain-### failed with error: com.vmware.vapi.std.errors.Error: Error (com.vmware.vapi.std.errors.error) => {
    messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {    id = com.vmware.vcIntegrity.lifecycle.TaskError.HealthCheckFailed,

vmware-vum-server.log reports EHP health check failures for Spherelet solution.

/var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log

YYYY-MM-DDTHH:MM:SS.442Z info vmware-vum-server[1308704] [Originator@6876 sub=EHP opID=########-####-####-####-############] [domain-###] [Unknown] [com.vmware.vcIntegrity.lifecycle.health.wcp.masterNode_status] returned status: NOT_OK

 
YYYY-MM-DDTHH:MM:SS839Z info vmware-vum-server[23925] [Originator@6876 sub=RemediateClusterTask opID=2########-####-####-####-############] [ApplyHelpers 688] CheckClusterHealth - calling EHP check completed. (cluster name = TEST-CLUSTER) - (perspective = 1) - (check result status = 3) - (check timeout = 4200)
YYYY-MM-DDTHH:MM:SS.839Z info vmware-vum-server[23925] [Originator@6876 sub=RemediateClusterTask opID=########-####-####-####-############] [ApplyHelpers 711] CheckClusterHealth - check -  issues size = 0 - status = 0 -  check = com.vmware.vpxd.healthPerspectives.ready_for_apply.ha  name = HA image constraints check - description = This check verifies that the HA constraints in the image spec about to be applied are set app
ropriately.
YYYY-MM-DDTHH:MM:SS.839Z info vmware-vum-server[23925] [Originator@6876 sub=RemediateClusterTask opID=########-####-####-####-############] [ApplyHelpers 711] CheckClusterHealth - check -  issues size = 0 - status = 0 -  check = com.vmware.vpxd.healthPerspectives.ready_for_apply.iofilter  name = IoFilter constraints check - description = This check verifies that iofilter constraints in the desrired spec about to be applied are set appropriately.
YYYY-MM-DDTHH:MM:SS.839Z info vmware-vum-server[23925] [Originator@6876 sub=RemediateClusterTask opID=########-####-####-####-############] [ApplyHelpers 711] CheckClusterHealth - check -  issues size = 1 - status = 3 -  check = com.vmware.vcIntegrity.lifecycle.health.wcp.masterNode_status  name = Supervisor control plane readiness check. - description = Check config and kubernetes status for Supervisor control plane readiness.
YYYY-MM-DDTHH:MM:SS.839Z info vmware-vum-server[23925] [Originator@6876 sub=RemediateClusterTask opID=########-####-####-####-############] [ApplyHelpers 711] CheckClusterHealth - check -  issues size = 0 - status = 0 -  check = com.vmware.vcIntegrity.lifecycle.health.wcp.sphereletSolution_status  name = Spherelet solution check. - description = Check if spherelet solution is set in the desired image of the cluster.
YYYY-MM-DDTHH:MM:SS.839Z info vmware-vum-server[23925] [Originator@6876 sub=RemediateClusterTask opID=########-####-####-####-############] [ApplyHelpers 711] CheckClusterHealth - check -  issues size = 0 - status = 0 -  check = com.vmware.nsxt.Mp_esx_version_check  name = Check for validating NSX-T version in the cluster desired state - description = This check validates whether the NSX-T component version in the cluster desired state is less than or equal to NSX-T Manager version.
YYYY-MM-DDTHH:MM:SS.839Z info vmware-vum-server[23925] [Originator@6876 sub=RemediateClusterTask opID=########-####-####-####-############] [ApplyHelpers 739] CheckClusterHealth - health check error - (cluster id = domain-###) - (cluster name = TEST-CLUSTER) - (perspective = 1) - (status = 3)
YYYY-MM-DDTHH:MM:SS.840Z info vmware-vum-server[23925] [Originator@6876 sub=RemediateClusterTask] [Task, 505] Task:com.vmware.vcIntegrity.lifecycle.RemediateClusterTask ID:########-####-####-####-############. Finalizing Task
YYYY-MM-DDTHH:MM:SS.840Z info vmware-vum-server[23925] [Originator@6876 sub=EHP] Deleting cached credentials for vapi session ID ########-####-####-####-############
YYYY-MM-DDTHH:MM:SS.840Z info vmware-vum-server[23925] [Originator@6876 sub=RemediateClusterTask] [ApplyHelpers 1266] Updating status with failure. -originator = vSphere Lifecycle Manager - retriable = false

 

Environment

VMware Cloud Foundation 9.X

Cause

Issue was caused by WCP Master Node Health Check failure.

WCP Supervisor cluster was in an error state.

To check the Supervisor cluster's health status: 

Logging into the vCenter Server-->Menu -> Workload Management-> Supervisors

Configuration error  
The control plane VM ############################### does not trust the load balancer's (Avi - https://##.##.##.##:443/login) certificate. Validate the Supervisor cluster load balancer's certificate authority configuration.

Resolution

  Refer to the below KB articles to fix the  AVI load balancer certificate issue:

  Supervisor control plane VM is unable to validate the AVI load balancer certificate

  WCP Supervisor Shows Error After Renewing AVI Load Balancer Certificate