Synchronize Inventory Versions fails with error : Duplicate Key Cannot connect to vCenter
search cancel

Synchronize Inventory Versions fails with error : Duplicate Key Cannot connect to vCenter

book

Article ID: 414537

calendar_today

Updated On:

Products

VMware SDDC Manager

Issue/Introduction

Synchronize Inventory Versions fails with error : Duplicate Key Cannot connect to vCenter.

less /var/log/vmware/vcf/operationsmanager/operationsmanager.log : 

YYYY-MM-DDT03:00:00.283+0300 DEBUG [vcf_om,68b63380b409584726735750cf275adf,e4db] [c.v.e.s.c.c.v.vsphere.VcManagerBase,pool-10-thread-1] Connecting to https://VCFQDN:443/sdk
YYYY-MM-DDT03:00:00.287+0300 DEBUG [vcf_om,68b63380b409584726735750cf275adf,e4db] [c.v.v.s.t.DynamicTrustManager,pool-10-thread-1] Checking validity of certificate chain CN=VCFQDN, OU=VMware Engineering, O=VMware, L=Palo Alto, ST=California, C=US
YYYY-MM-DDT03:00:00.288+0300 DEBUG [vcf_om,68b63380b409584726735750cf275adf,e4db] [c.v.v.s.t.DynamicTrustManager,pool-10-thread-1] Certificate chain CN=VCFQDN, OU=VMware Engineering, O=VMware, L=Palo Alto, ST=California, C=US is valid
YYYY-MM-DDT03:00:00.292+0300 WARN  [vcf_om,68b63380b409584726735750cf275adf,e4db] [c.v.v.v.c.h.i.HttpProtocolBindingBase,pool-10-thread-1] Asynchronous execution requested but no Executor configured. The request will be executed as synchronous one.
YYYY-MM-DDT03:00:03.624+0300 ERROR [vcf_om,68b63380b409584726735750cf275adf,e4db] [c.v.e.s.c.c.v.vsphere.VsphereClient,pool-10-thread-1] Failed to connect to https://VCFQDN:443/sdk
java.util.concurrent.ExecutionException: (vim.fault.InvalidLogin) {
   faultCause = null,
   faultMessage = null
}        at com.vmware.vim.vmomi.core.impl.BlockingFuture.get(BlockingFuture.java:81)
        at com.vmware.evo.sddc.common.client.vmware.vsphere.VsphereClient.<init>(VsphereClient.java:121)
Caused by: com.vmware.vim.binding.vim.fault.InvalidLogin: Cannot complete login due to an incorrect user name or password.
YYYY-MM-DDT03:00:03.639+0300 WARN  [vcf_om,68b63380b409584726735750cf275adf,e4db] [c.v.e.s.c.c.v.vsphere.VsphereClient,pool-10-thread-1] Error logging out of session
com.vmware.vim.binding.vim.fault.NotAuthenticated: The session is not authenticated.
        at jdk.internal.reflect.GeneratedConstructorAccessor2000.newInstance(Unknown Source)
YYYY-MM-DDT03:00:03.640+0300 ERROR [vcf_om,68b63380b409584726735750cf275adf,e4db] [c.v.e.s.c.c.v.vsphere.VcManagerBase,pool-10-thread-1] Cannot complete login due to incorrect credentials: VCFQDN, administrator@WRONGSSODOMAIN
YYYY-MM-DDT03:00:03.640+0300 ERROR [vcf_om,68b63380b409584726735750cf275adf,e4db] [c.v.v.i.s.s.impl.EsxiVersionSync,pool-10-thread-1] Exception when constructing version diff for host(s) assigned to vcenter VCFQDN
java.util.concurrent.ExecutionException: (vim.fault.InvalidLogin) {
   faultCause = null,
   faultMessage = null
}
YYYY-MM-DDT03:00:06.936+0300 DEBUG [vcf_om,68b63380b409584726735750cf275adf,e4db] [c.v.v.v.s.i.VersionSyncServiceImpl,pool-10-thread-1] Validating the result of the inventory sync: {"entitiesToSync":["VCENTER","ESXI","NSXT_CLUSTER"],"entitiesToUpdate":[],"completedInventorySyncTasks":[{"entity":"ESXI","syncStatus":"FAILED","errors":[{"errorCode":"ESX_RESOURCE_VERSION_FETCH_FAILED","errorMessage":"Cannot connect to vCenter VCFQDN of domain WLDDOMAINNAME to complete inventory sync.","errorType":"ERROR","cause":"java.util.concurrent.ExecutionException: (vim.fault.InvalidLogin) {\n   faultCause \u003d null,\n   faultMessage \u003d null\n}","remediation":"Please ensure the vCenter VCFQDN is up and running."}]},{"entity":"NSXT_CLUSTER","syncStatus":"SUCCEEDED","errors":[]},{"entity":"VCENTER","syncStatus":"FAILED","errors":[{"errorCode":"VC_RESOURCE_VERSION_FETCH_FAILED","errorMessage":"Cannot connect to vCenter VCFQDN of domain WLDDOMAINNAME to complete inventory sync.","errorType":"ERROR","cause":"java.lang.RuntimeException: Exception occurred during vAPI invocation: java.util.concurrent.ExecutionException: com.vmware.vapi.std.errors.Unauthenticated: Unauthenticated (com.vmware.vapi.std.errors.unauthenticated) \u003d\u003e {\n    messages \u003d [],\n    data \u003d \u003cnull\u003e,\n    errorType \u003d UNAUTHENTICATED,\n    challenge \u003d SIGN realm\u003d\"THUMBPRINT\",sts\u003d\"https://VCFQDN/sts/STSService/vsphere.local\", Basic realm\u003d\"vCenter\"\n}","remediation":"Please ensure the vCenter VCFQDN is up and SSH connectivity is available. Reboot/Power up the vCenter if required."}]}],"failedInventorySyncTasks":[{"entity":"ESXI","syncStatus":"FAILED","errors":[{"errorCode":"ESX_RESOURCE_VERSION_FETCH_FAILED","errorMessage":"Cannot connect to vCenter VCFQDN of domain WLDDOMAINNAME to complete inventory sync.","errorType":"ERROR","cause":"java.util.concurrent.ExecutionException: (vim.fault.InvalidLogin) {\n   faultCause \u003d null,\n   faultMessage \u003d null\n}","remediation":"Please ensure the vCenter VCFQDN is up and running."}]},{"entity":"VCENTER","syncStatus":"FAILED","errors":[{"errorCode":"VC_RESOURCE_VERSION_FETCH_FAILED","errorMessage":"Cannot connect to vCenter VCFQDN of domain WLDDOMAINNAME to complete inventory sync.","errorType":"ERROR","cause":"java.lang.RuntimeException: Exception occurred during vAPI invocation: java.util.concurrent.ExecutionException: com.vmware.vapi.std.errors.Unauthenticated: Unauthenticated (com.vmware.vapi.std.errors.unauthenticated) \u003d\u003e {\n    messages \u003d [],\n    data \u003d \u003cnull\u003e,\n    errorType \u003d UNAUTHENTICATED,\n    challenge \u003d SIGN realm\u003d\"THUMBPRINT\",sts\u003d\"https://VCFQDN/sts/STSService/vsphere.local\", Basic realm\u003d\"vCenter\"\n}","remediation":"Please ensure the vCenter VCFQDN is up and SSH connectivity is available. Reboot/Power up the vCenter if required."}]}]}
YYYY-MM-DDT03:00:06.939+0300 ERROR [vcf_om,68b63380b409584726735750cf275adf,e4db] [c.v.v.v.s.s.VersionSyncScheduler,pool-10-thread-1] Failed to validate/trigger inventory version sync
org.springframework.retry.ExhaustedRetryException: Cannot locate recovery method
Caused by: java.lang.IllegalStateException: Duplicate key Cannot connect to vCenter VCFQDN of domain WLDDOMAINNAME to complete inventory sync. (attempted merging values Please ensure the vCenter VCFQDN is up and running. and Please ensure the vCenter VCFQDN is up and SSH connectivity is available. Reboot/Power up the vCenter if required.)

 

To check the credentials : 

psql -h localhost -U postgres -d platform -c "select * from credential where entitytype='PSC' and credentialtype='SSO';

The output may display wrong SSO Domain at the bottom. 

 

Environment

SDDC Manager

Cause

/v1/credentials API picks the credentials in the reverse order causing the issue. 

Resolution

Engineering are aware of the reported issue and the fix in the code is implemented for the next software release.

Powered by Wolken Software