Removal of Workload Domain from SDDC Manager fails at task "Disable vSphere Cluster Services on cluster".

Products

• VMware SDDC Manager

Issue/Introduction

While attempting to remove a Workload Domain via the SDDC UI, the workflow fails at the "Disable vSphere Cluster Services on cluster" task with an error -

Failed to disable vSphere Cluster Services on cluster

Check vCenter connectivity

Cannot complete login due to an incorrect user name or password

It has been verified on the SDDC Manager UI > Security >  Password Management > vCenter page that the SDDC Manager service account configured on vCenter has a status of "Active".

This error is observed in the log /var/log/vmware/vcf/domainmanager/domainmanager.log -

YYYY-MM-DDThh:mm:ss.sss+0000 DEBUG [vcf_dm,xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx,xxxx] [c.v.e.s.o.c.c.ContractParamBuilder,dm-exec-14]  Contract task Disable vSphere Cluster Services input: {"vcEndpoint":{"address":"vcenter.fqdn","port":0,"username":"[email protected]","password":"*****"},"clusterMoids":["domain-c9"],"validateVclsVms":true,"ignoreVcenterConnectionIssues":true}
YYYY-MM-DDThh:mm:ss.sss+0000 ERROR [vcf_dm,xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx,xxxx] [c.v.e.s.c.c.v.vsphere.VsphereClient,dm-exec-14]  Failed to connect to https://vcenter.fqdn/sdk  
java.util.concurrent.ExecutionException: (vim.fault.InvalidLogin) {  
   faultCause = null,  
   faultMessage = null
Caused by: com.vmware.vim.binding.vim.fault.InvalidLogin: Cannot complete login due to an incorrect user name or password.

Environment

VCF 5.2.1

Cause

The password for the SDDC Manager service account configured on the vCenter had rotated between when the workflow started and when the task to "Disable vSphere Cluster Services on cluster" was reached.

The workflow was using an old password, and as such authentication was not possible.

Resolution

Implement the steps below to resolve the issue -

1. Take a snapshot of the SDDC Manager VM, without memory. This step is mandatory.
2. Use this article to retrieve what the current password for the SDDC Manager service account configured on the vCenter is - https://knowledge.broadcom.com/external/article?articleId=327195
3. Once the current password has been retrieved - export the failed workflow, update the password in it, and inject the updated workflow back into the SDDC using this article - https://knowledge.broadcom.com/external/article?articleNumber=314620
4. Retry the failed workflow, it is now expected to succeed given that it is now using the correct current password for the SDDC Manager service account configured on the vCenter.