Synchronize Inventory Versions fails with error : Duplicate Key Cannot connect to vCenter
Article ID: 414537
Updated On:
Products
VMware SDDC Manager / VCF Installer
Issue/Introduction
Synchronize Inventory Versions fails with error: Duplicate Key Cannot connect to vCenter.
less /var/log/vmware/vcf/operationsmanager/operationsmanager.log :
YYYY-MM-DDT03:00:00.283+0300 DEBUG [vcf_om,68b63380b409584726735750cf275adf,e4db] [c.v.e.s.c.c.v.vsphere.VcManagerBase,pool-10-thread-1] Connecting to https://VCFQDN:443/sdk
YYYY-MM-DDT03:00:00.287+0300 DEBUG [vcf_om,68b63380b409584726735750cf275adf,e4db] [c.v.v.s.t.DynamicTrustManager,pool-10-thread-1] Checking validity of certificate chain CN=VCFQDN, OU=VMware Engineering, O=VMware, L=Palo Alto, ST=California, C=US
YYYY-MM-DDT03:00:00.288+0300 DEBUG [vcf_om,68b63380b409584726735750cf275adf,e4db] [c.v.v.s.t.DynamicTrustManager,pool-10-thread-1] Certificate chain CN=VCFQDN, OU=VMware Engineering, O=VMware, L=Palo Alto, ST=California, C=US is valid
YYYY-MM-DDT03:00:00.292+0300 WARN [vcf_om,68b63380b409584726735750cf275adf,e4db] [c.v.v.v.c.h.i.HttpProtocolBindingBase,pool-10-thread-1] Asynchronous execution requested but no Executor configured. The request will be executed as synchronous one.
YYYY-MM-DDT03:00:03.624+0300 ERROR [vcf_om,68b63380b409584726735750cf275adf,e4db] [c.v.e.s.c.c.v.vsphere.VsphereClient,pool-10-thread-1] Failed to connect to https://VCFQDN:443/sdk
java.util.concurrent.ExecutionException: (vim.fault.InvalidLogin) {
faultCause = null,
faultMessage = null
} at com.vmware.vim.vmomi.core.impl.BlockingFuture.get(BlockingFuture.java:81)
at com.vmware.evo.sddc.common.client.vmware.vsphere.VsphereClient.<init>(VsphereClient.java:121)
Caused by: com.vmware.vim.binding.vim.fault.InvalidLogin: Cannot complete login due to an incorrect user name or password.
YYYY-MM-DDT03:00:03.639+0300 WARN [vcf_om,68b63380b409584726735750cf275adf,e4db] [c.v.e.s.c.c.v.vsphere.VsphereClient,pool-10-thread-1] Error logging out of session
com.vmware.vim.binding.vim.fault.NotAuthenticated: The session is not authenticated.
at jdk.internal.reflect.GeneratedConstructorAccessor2000.newInstance(Unknown Source)
YYYY-MM-DDT03:00:03.640+0300 ERROR [vcf_om,68b63380b409584726735750cf275adf,e4db] [c.v.e.s.c.c.v.vsphere.VcManagerBase,pool-10-thread-1] Cannot complete login due to incorrect credentials: VCFQDN, administrator@WRONGSSODOMAIN
YYYY-MM-DDT03:00:03.640+0300 ERROR [vcf_om,68b63380b409584726735750cf275adf,e4db] [c.v.v.i.s.s.impl.EsxiVersionSync,pool-10-thread-1] Exception when constructing version diff for host(s) assigned to vcenter VCFQDN
java.util.concurrent.ExecutionException: (vim.fault.InvalidLogin) {
faultCause = null,
faultMessage = null
}
YYYY-MM-DDT03:00:06.936+0300 DEBUG [vcf_om,68b63380b409584726735750cf275adf,e4db] [c.v.v.v.s.i.VersionSyncServiceImpl,pool-10-thread-1] Validating the result of the inventory sync: {"entitiesToSync":["VCENTER","ESXI","NSXT_CLUSTER"],"entitiesToUpdate":[],"completedInventorySyncTasks":[{"entity":"ESXI","syncStatus":"FAILED","errors":[{"errorCode":"ESX_RESOURCE_VERSION_FETCH_FAILED","errorMessage":"Cannot connect to vCenter VCFQDN of domain WLDDOMAINNAME to complete inventory sync.","errorType":"ERROR","cause":"java.util.concurrent.ExecutionException: (vim.fault.InvalidLogin) {\n faultCause \u003d null,\n faultMessage \u003d null\n}","remediation":"Please ensure the vCenter VCFQDN is up and running."}]},{"entity":"NSXT_CLUSTER","syncStatus":"SUCCEEDED","errors":[]},{"entity":"VCENTER","syncStatus":"FAILED","errors":[{"errorCode":"VC_RESOURCE_VERSION_FETCH_FAILED","errorMessage":"Cannot connect to vCenter VCFQDN of domain WLDDOMAINNAME to complete inventory sync.","errorType":"ERROR","cause":"java.lang.RuntimeException: Exception occurred during vAPI invocation: java.util.concurrent.ExecutionException: com.vmware.vapi.std.errors.Unauthenticated: Unauthenticated (com.vmware.vapi.std.errors.unauthenticated) \u003d\u003e {\n messages \u003d [],\n data \u003d \u003cnull\u003e,\n errorType \u003d UNAUTHENTICATED,\n challenge \u003d SIGN realm\u003d\"THUMBPRINT\",sts\u003d\"https://VCFQDN/sts/STSService/vsphere.local\", Basic realm\u003d\"vCenter\"\n}","remediation":"Please ensure the vCenter VCFQDN is up and SSH connectivity is available. Reboot/Power up the vCenter if required."}]}],"failedInventorySyncTasks":[{"entity":"ESXI","syncStatus":"FAILED","errors":[{"errorCode":"ESX_RESOURCE_VERSION_FETCH_FAILED","errorMessage":"Cannot connect to vCenter VCFQDN of domain WLDDOMAINNAME to complete inventory sync.","errorType":"ERROR","cause":"java.util.concurrent.ExecutionException: (vim.fault.InvalidLogin) {\n faultCause \u003d null,\n faultMessage \u003d null\n}","remediation":"Please ensure the vCenter VCFQDN is up and running."}]},{"entity":"VCENTER","syncStatus":"FAILED","errors":[{"errorCode":"VC_RESOURCE_VERSION_FETCH_FAILED","errorMessage":"Cannot connect to vCenter VCFQDN of domain WLDDOMAINNAME to complete inventory sync.","errorType":"ERROR","cause":"java.lang.RuntimeException: Exception occurred during vAPI invocation: java.util.concurrent.ExecutionException: com.vmware.vapi.std.errors.Unauthenticated: Unauthenticated (com.vmware.vapi.std.errors.unauthenticated) \u003d\u003e {\n messages \u003d [],\n data \u003d \u003cnull\u003e,\n errorType \u003d UNAUTHENTICATED,\n challenge \u003d SIGN realm\u003d\"THUMBPRINT\",sts\u003d\"https://VCFQDN/sts/STSService/vsphere.local\", Basic realm\u003d\"vCenter\"\n}","remediation":"Please ensure the vCenter VCFQDN is up and SSH connectivity is available. Reboot/Power up the vCenter if required."}]}]}
YYYY-MM-DDT03:00:06.939+0300 ERROR [vcf_om,68b63380b409584726735750cf275adf,e4db] [c.v.v.v.s.s.VersionSyncScheduler,pool-10-thread-1] Failed to validate/trigger inventory version sync
org.springframework.retry.ExhaustedRetryException: Cannot locate recovery method
Caused by: java.lang.IllegalStateException: Duplicate key Cannot connect to vCenter VCFQDN of domain WLDDOMAINNAME to complete inventory sync. (attempted merging values Please ensure the vCenter VCFQDN is up and running. and Please ensure the vCenter VCFQDN is up and SSH connectivity is available. Reboot/Power up the vCenter if required.)
This error can occur when performing pre-checks for upgrades with SDDC.
Environment
SDDC Manager
Cause
The system's /v1/credentials API picks the credentials in the reverse order, resulting in the wrong service account trying to connect to the wrong vCenter.
This can result in environments where the noted vCenter has a different SSO domain than the noted service account in the error.
Resolution
This issue is fixed in SDDC Manager v5.2.3.
If you encounter this issue, reach to to VMware by Broadcom Technical Support referencing this KB.
Feedback
Yes
No
Powered by
