"Synchronize Inventory Versions" task failed in the SDDC Manager UI.
search cancel

"Synchronize Inventory Versions" task failed in the SDDC Manager UI.

book

Article ID: 439995

calendar_today

Updated On:

Products

VMware SDDC Manager

Issue/Introduction

  • Inventory version synchronization failed with: Cannot connect to vCenter <vCenter_ID> of domain to complete inventory sync.

  • /var/log/vmware/vcf/operationsmanager/operationsmanager.log

YYYY-MM-DDT03:04:47.750+0000 DEBUG [vcf_om,69fd52ca8a266af9bc35cacc880e####,69f6] [c.v.v.v.s.i.VersionSyncServiceImpl,pool-3-thread-1] Validating the result of the inventory sync:{
"entitiesToSync":["VCENTER","NSXT_CLUSTER","ESXI"],"entitiesToUpdate":[],"completedInventorySyncTasks":[{"entity":"NSXT_CLUSTER","syncStatus":"SUCCEEDED","errors":[]},{"entity":"VCENTER","syncStatus":"SUCCEEDED","errors":[]},{"entity":"ESXI","syncStatus":"FAILED","errors":[{"errorCode":"ESX_RESOURCE_VERSION_FETCH_FAILED","errorMessage":"Cannot connect to vCenter a8fb5783-be29-4166-####-######### of domain <domain_name> to complete inventory sync.","errorType":"ERROR","cause":"","remediation":"Please ensure the vCenter a8fb5783-be29-4166-####-######### is up and running."}]}],"failedInventorySyncTasks":[{"entity":"ESXI","syncStatus":"FAILED","errors":[{"errorCode":"ESX_RESOURCE_VERSION_FETCH_FAILED","errorMessage":"Cannot connect to vCenter a8fb5783-be29-4166-####-######### of domain <domain_name> to complete inventory sync.","errorType":"ERROR","cause":"","remediation":"Please ensure the vCenter a8fb5783-be29-4166-####-######### is up and running."}]}]}
YYYY-MM-DDT03:04:47.751+0000 ERROR [vcf_om,69fd52ca8a266af9bc35cacc880e####,69f6] [c.v.e.s.o.model.error.ErrorFactory,pool-3-thread-1] [T9DP6I] VERSION_SYNC_FAILED Inventory version synchronization failed. Please try to fix the cause for the issue and retry the operation.
com.vmware.evo.sddc.common.core.error.InvalidStateException: Inventory version synchronization failed. Please try to fix the cause for the issue and retry the operation.
at com.vmware.vcf.versionsync.services.impl.VersionSyncServiceImpl.validateInventorySyncStatus(VersionSyncServiceImpl.java:91)
at com.vmware.vcf.versionsync.services.impl.VersionSyncServiceImpl.syncInventory(VersionSyncServiceImpl.java:43)
at com.vmware.vcf.versionsync.services.action.SyncInventoryVersionsAction.execute(SyncInventoryVersionsAction.java:44)
at com.vmware.vcf.versionsync.services.action.SyncInventoryVersionsAction.execute(SyncInventoryVersionsAction.java:21)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:62)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:159)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:144)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:400)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:520)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:124)
at jdk.internal.reflect.GeneratedMethodAccessor2517.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:85)
at com.google.common.eventbus.Subscriber.lambda$dispatchEvent$0(Subscriber.java:71)
at com.vmware.vcf.common.tracing.TraceRunnable.run(TraceRunnable.java:59)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: com.vmware.evo.sddc.common.core.error.InvalidStateException: Inventory version synchronization failed with: Cannot connect to vCenter a8fb5783-be29-4166-####-#########
of domain <domain_name> to complete inventory sync.
at com.vmware.vcf.versionsync.services.impl.VersionSyncServiceImpl.validateInventorySyncStatus(VersionSyncServiceImpl.java:89)
... 18 common frames omitted

 

 

 

Environment

VCF 5.2.x

Cause

The presence of host IDs in the host table of the platform database with a status of 'ERROR', coupled with their absence from the vCenter inventory, directly correlates with the "synchronize Inventory versions failed" messages observed in the Operations Manager logs.

Resolution

Manually remove the stale entries of the hosts which are in ERROR status in SDDC platform database and trigger a manual inventory synchronization

Note : Please take offline snapshot of the SDDC Manager before proceeding with the below steps.

1. Login to the SSH of the SDDC Manager with vcf and switch to root user using su -

2. Connect to the platform database using the command : psql -h localhost -U postgres -d platform

3. To query the host id  which are in ERROR status execute the command : select id, hostname, status from host where status='ERROR';

Sample Output : 

4 . Remove the host ID from all related tables.

delete from host where id='<host_id>';
delete from host_and_cluster where host_id='<host_id>';
delete from host_and_domain where host_id='<host_id>';
delete from host_and_network_pool where host_id='<host_id>';
delete from host_and_vcenter where host_id='<host_id>';

5 . Restart the SDDC services using the command : 

/opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

6 . Execute a manual inventory sync as per Synchronize inventory versions after out-of-band upgrade in a VMware Cloud Foundation Environment

Powered by Wolken Software