General upgrade readiness precheck and inventory-sync fails with isolated domains in SDDC

Products

• VMware SDDC Manager

Issue/Introduction

/var/log/vmware/vcf/operationsmanager/operationsmanger.log shows a failure during the upgrade precheck for the "out-of-band-upgrade-validator"

YYYY-MM-DDT##:##:##+0000 ERROR [vcf_om,<UUID>,<####>] [c.v.v.b.p. t. v. SimpleNativeValidator, precheck-validation10] Native validator with name 'out-of-band-upgrade-validator' failed internally. Reference token: <tokenID>
java. lang. IllegalStateException: Duplicate key Cannot connect to vCenter <vcenterFQDN> of domain <domain name> to complete inventory sync. (attempted merging values Please ensure the vCenter <vcenterFQDN> is up and SSH connectivity is available.  Reboot/Power up the vCenter and Please ensure the vCenter n-rall-svb-vc.vcf. fcbint.net is up and running.)

operationsmanager.log shows a similar error during a failed inventory sync task

YYYY-MM-DDT##:##:##+0000 ERROR [vcf_om,<UUID>,<####>] [c.v.e.s.c.c.v.vsphere.VcManagerBase,pool-3-thread-13] Cannot complete login due to incorrect credentials: <vcenterFQDN>, [email protected].
YYYY-MM-DDT##:##:##+0000 ERROR [vcf_om,<UUID>,<####>] [c.v.v.i.s.s.impl.EsxiVersionSync,pool-3-thread-13] Exception when constructing version diff for host(s) assigned to vcenter <vcenterFQDN>
java.util.concurrent.ExecutionException: (vim.fault.InvalidLogin) {
   faultCause = null,
   faultMessage = null
}
        ....

Caused by: com.vmware.vim.binding.vim.fault.InvalidLogin: Cannot complete login due to an incorrect user name or password.

Environment

VMware SDDC Manager 5.x

Cause

General upgrade precheck and inventory-sync fails with isolated domains due to SDDC trying to use the wrong [email protected] credentials (which have a different password) to authenticate to each vCenter.

Resolution

Update all the domains to use the same same [email protected] password, so that the wrong password can't be passed around during the inventory sync and out-of-band validation.

Managing Passwords in VMware Cloud Foundation

Additional Information

Engineering is working on a fix for this issue to be released in a future update.