Retry of a failed domain workflow fails with error VCENTER_CONNECTION_FAILED on SDDC UI if passwords are rotated between the failure and retry

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• Any day-N domain manager operations related to the creation, deletion, and scaling of workload domains that had failed previously give an error on retry VCENTER_CONNECTION_FAILED Failed running operation with vSphere
• lookup_password entry of SSO credentials will match the credentials used to login to vCenter
• /var/log/vmware/vcf/domainmanager/domainmanager.log logs show 

2022-09-06T07:10:44.308+0000 ERROR [vcf_dm,xxxxxxxxxxxxxxxx,xxxx] [c.v.e.s.c.c.v.vsphere.VsphereClient,dm-exec-10] Failed to connect to https://<vCenterFQDN>:443/sdk java.util.concurrent.ExecutionException: (vim.fault.InvalidLogin) {

  faultCause = null,

  faultMessage = null

}
2022-09-06T07:10:44.319+0000 WARN [vcf_dm,xxxxxxxxxxxxxxxx,xxxx] [c.v.e.s.c.c.v.vsphere.VsphereClient,dm-exec-10] Error logging out of session com.vmware.vim.binding.vim.fault.NotAuthenticated: The session is not authenticated.


2022-09-06T07:11:00.817+0000 ERROR [vcf_dm,xxxxxxxxxxxxxxxx,xxxx] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-12] [LI97GE] VCENTER_CONNECTION_FAILED Failed running operation with domainnamne @ vCenterFQDN

com.vmware.evo.sddc.common.core.error.LocalizableRuntimeException: Failed running operation with domainname @ vCenterFQDN

    at com.vmware.vcf.common.vsphere.plugins.action.VsphereUser$VsphereRunner.use(VsphereUser.java:156)

    at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:124)

Caused by: java.util.concurrent.ExecutionException: (vim.fault.InvalidLogin) {

  faultCause = null,

  faultMessage = null

}

Environment

VMware Cloud Foundation 

Resolution

Retry of the task would fail and a new task needs to be initiated to fix this issue as the password is rotated to a different one than the previous task has cached.

Example: If its expanding cluster workflow the host will have to be decommissioned, re-imaged and commissioned back. Post that fresh workflow needs to be started

Additional Information

Impact/Risks:
Retry of a critical Domainmanager operation fails with authentication errors