Expanding a stretch cluster in SDDC Manager fails at Remediate ESXi Host(s) to be Compliant with Cluster's Image

Products

VMware SDDC Manager / VCF Installer VMware Cloud Foundation

Issue/Introduction

Expanding a stretch cluster in SDDC Manager fails at Remediate ESXi Host(s) to be Compliant with Cluster's Image

Error in SDDC Manager UI

Progress Messages	Applying personality to cluster domain-c# failed with error: ApplyStatus (com.vmware.esx.settings.clusters.software.apply_status) => { status = SKIPPED, progress = <null>, startTime = yyyy-mm-ddThh:mm:ss.325Z[GMT], endTime = yyyy-mm-ddThh:mm:ss.385Z[GMT], notifications = Notifications (com.vmware.esx.settings.notifications) => { info = <null>, warnings = <null>, errors = <null> } } Applying personality to cluster domain-c# failed with error: ApplyStatus (com.vmware.esx.settings.clusters.software.apply_status) => { status = SKIPPED, progress = <null>, startTime = yyyy-mm-ddThh:mm:ss.859Z[GMT], endTime = yyyy-mm-ddThh:mm:ss.667Z[GMT], notifications = Notifications (com.vmware.esx.settings.notifications) => { info = <null>, warnings = <null>, errors = <null> } } Applying personality to cluster domain-c# failed with error: ApplyStatus (com.vmware.esx.settings.clusters.software.apply_status) => { status = SKIPPED, progress = <null>, startTime = yyyy-mm-ddThh:mm:ss.427Z[GMT], endTime = yyyy-mm-ddThh:mm:ss.731Z[GMT], notifications = Notifications (com.vmware.esx.settings.notifications) => { info = <null>, warnings = <null>, errors = <null> } } Applying personality to cluster domain-c# failed with error: ApplyStatus (com.vmware.esx.settings.clusters.software.apply_status) => { status = SKIPPED, progress = <null>, startTime = yyyy-mm-ddThh:mm:ss.084Z[GMT], endTime = yyyy-mm-ddThh:mm:ss.155Z[GMT], notifications = Notifications (com.vmware.esx.settings.notifications) => { info = <null>, warnings = <null>, errors = <null> } }
	Failed to validate since personality was already applied on cluster domain-c#.
Error	Message: Applying personality to cluster domain-c# failed with error: ApplyStatus (com.vmware.esx.settings.clusters.software.apply_status) => { status = SKIPPED, progress = <null>, startTime = yyyy-mm-ddThh:mm:ss.427Z[GMT], endTime = yyyy-mm-ddThh:mm:ss.731Z[GMT], notifications = Notifications (com.vmware.esx.settings.notifications) => { info = <null>, warnings = <null>, errors = <null> } } Remediation Message: Reference Token: 6####6
Error	Message: Failed to validate since personality was already applied on cluster domain-c#. Remediation Message: Reference Token: G####D Cause: Exception occurred during vAPI invocation: java.util.concurrent.ExecutionException: com.vmware.vapi.std.errors.Unauthenticated: Unauthenticated (com.vmware.vapi.std.errors.unauthenticated) => { messages = [], data = <null>, errorType = UNAUTHENTICATED, challenge = SIGN realm="A6:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:BA",sts="https://vcenter.example.com/sts/STSService/vsphere.local", Basic realm="vCenter" }

Error in /var/log/vmware/vcf/domainmanager/domainmanager.log

ERROR [vcf_dm,68#######################45,e448] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-25]  [6425LR] VLCM_REMEDIATE_PERSONALITY_FAILED Applying personality to cluster domain-c# failed with error: ApplyStatus (com.vmware.esx.settings.clusters.software.apply_status) => {
    status = SKIPPED,
    progress = <null>,
    startTime = yyyy-mm-ddThh:mm:ss.084Z[GMT],
    endTime = yyyy-mm-ddThh:mm:ss.155Z[GMT],
    notifications = Notifications (com.vmware.esx.settings.notifications) => {
        info = <null>,
        warnings = <null>,
        errors = <null>
    }
}
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Applying personality to cluster domain-c9 failed with error: ApplyStatus (com.vmware.esx.settings.clusters.software.apply_status) => {
    status = SKIPPED,
    progress = <null>,
    startTime = yyyy-mm-ddThh:mm:ss.084Z[GMT],
    endTime = yyyy-mm-ddThh:mm:ss.155Z[GMT],
    notifications = Notifications (com.vmware.esx.settings.notifications) => {
        info = <null>,
        warnings = <null>,
        errors = <null>
    }
}

ESX hosts are in compliance with the image in the vSphere Cluster

Environment

VMware Cloud Foundation 9.x

Cause

vCenter compliance checks are taking a significant amount of time to complete, the workflow is not returning to the domainmanager to continue the add host workflow. This causes an exception when remediating the cluster.

Resolution

Please contact Broadcom Support if you can not upgrade and would like to preform a workaround of this issue. Refer - Contact Broadcom support

Additional Information

Note: The error could persists the resolution steps above if the password for the SDDC Manager service account configured on the vCenter had rotated between when the workflow started and when the task to "Remediate ESXi Host(s) to be Compliant with Cluster's Image Failed" was reattempted.

In that case as the workflow is using the old password, the authentication to the vCenter will not be possible.

This can be verified in the task logs:

/var/log/vmware/vcf/domainmanager/domainmanager.log

task Remediate ESXi Host(s) to be Compliant with Cluster's Image input: {"vcEndpoint":{"address":"<vc_fqdn>","port":0,"username":"svc-vcf-<ssdc_fqdn>-<vc_fqdn>@vsphere.local","password":"*****"},"clusterMoids":["domain-cX"],"validateVclsVms":true,"ignoreVcenterConnectionIssues":true}
ERROR [vcf_dm,<task_id>] [c.v.e.s.c.c.v.vsphere.VsphereClient,dm-exec-XX]  Failed to connect to https://<vc_fqdn>/sdk  
java.util.concurrent.ExecutionException: (vim.fault.InvalidLogin) {  
   faultCause = null,  
   faultMessage = null
Caused by: com.vmware.vim.binding.vim.fault.InvalidLogin: Cannot complete login due to an incorrect user name or password.

In that scenario implement the steps below to resolve the issue -

Take a snapshot of the SDDC Manager VM, without memory.
Use this article to retrieve what the current password for the SDDC Manager service account configured on the vCenter is - https://knowledge.broadcom.com/external/article?articleId=327195
Once the current password has been retrieved - export the failed workflow, update the password in it, and inject the updated workflow back into the SDDC using this article - https://knowledge.broadcom.com/external/article?articleNumber=314620
Retry the failed workflow, it is now expected to succeed given that it is now using the correct current password for the SDDC Manager service account configured on the vCenter.