Configuring vSAN stretched Cluster fails with error "Failed to update and re-apply vSAN storage policy VsanDefaultProfile" on SDDC Manager

Products

• VCF Operations

Issue/Introduction

• The 'stretched cluster deployment' task on the SDDC Manager fails at subtask 'Update vSAN Storage Profile' with the following error:
  
  Failed to update and re-apply vSAN storage policy VsanDefaultProfile
Message: Failed to update and re-apply vSAN storage policy VsanDefaultProfile
Remediation Message:
Reference Token: ######
Cause: (vim.fault.InvalidLogin) { faultCause = null, faultMessage = null } Cannot
complete login due to an incorrect user name or password.


• On the SDDC Manager, under /var/log/vmware/vcf/domainmanager/domainmanager.log:

YYYY-MM-DDTHH:MM:SS ERROR [vcf_dm,###########,####] [c.v.e.s.c.c.v.vsphere.VsphereClient,dm-exec-13]  Failed to connect to https://<vCenter_FQDN>:443/sdk as svc-<SDDC_Manager>-<vCenter>@vsphere.local
java.util.concurrent.ExecutionException: (vim.fault.InvalidLogin) {
   faultCause = null,
   faultMessage = null
}
Caused by: com.vmware.vim.binding.vim.fault.InvalidLogin: Cannot complete login due to an incorrect user name or password.


• The 'stretched cluster deployment' task was started and then triggered again later, with the password for the vCenter service account (svc-<SDDC_Manager>-<vCenter>@vsphere.local) being rotated in the interval between the two executions.

Environment

VMware Cloud Foundation 9.0.x

Cause

The failure occurs because the vCenter service account password was rotated after the stretched cluster deployment task had already been initiated.

When the task is later resumed, the active workflow continues to use the previously cached credentials, which are no longer valid, resulting in an authentication failure.

Resolution

Fetch the latest valid password stored in the SDDC Manager database and inject it into the parameters of the failed workflow by following the steps below:

1. Take a snapshot of SDDC Manager Virtual Machine.
2. Retrieve the password for account svc-<SDDC_Manager>-<vCenter>@vsphere.local. Refer Retrieve the service accounts credentials from SDDC Manager
3. Update the running workflow with the retrieved password for the above account. Refer KB Re-try an existing workflow by modifying the workflow spec file