VCF Inventory-Sync fails with a timeout on VxRail Manager

Products

• VMware SDDC Manager

Issue/Introduction

Manually testing the /v1/system to VxRail responds but takes longer than 90 secs to respond. 

time curl -k -s -X GET --user '[email protected]:XXXXX' https://########.#####.###/rest/vxm/v1/system | json_pp

• Replace ########.#####.### with the VxRail Manager FQDN
• Replace XXXXX with the administrator password

/var/log/vmware/vcf/operationsmanager/operationsmanager.log
======================================================================================
2025-03-12T19:43:44.846+0000 ERROR [vcf_om,67d1e2975ce6ccc4b639261f8d49e75c,fa33] [c.v.v.i.s.s.i.VxRailManagerVersionSync,pool-3-thread-3] Exception when constructing version diff for vxmanager(s) assigned to vcenter #######.#####.###
com.vmware.vcf.inventory.sync.service.exception.InventorySyncException: Unable to fetch VxRail Manager version for hostname #######.#####.### and user [email protected] to complete inventory sync.
        at com.vmware.vcf.inventory.sync.service.impl.VxRailManagerVersionSync.constructVersionDiffForVxManagers(VxRailManagerVersionSync.java:285)
        at com.vmware.vcf.inventory.sync.service.impl.VxRailManagerVersionSync.constructVersionDiff(VxRailManagerVersionSync.java:110)
        at com.vmware.vcf.inventory.sync.service.impl.InventorySyncServiceImpl.syncInventory(InventorySyncServiceImpl.java:131)
        at com.vmware.vcf.inventory.sync.service.impl.InventorySyncServiceImpl.syncInventory(InventorySyncServiceImpl.java:53)
        at com.vmware.vcf.versionsync.services.impl.VersionSyncServiceImpl.syncInventory(VersionSyncServiceImpl.java:38)
        at com.vmware.vcf.versionsync.services.action.SyncInventoryVersionsAction.execute(SyncInventoryVersionsAction.java:44)
        at com.vmware.vcf.versionsync.services.action.SyncInventoryVersionsAction.execute(SyncInventoryVersionsAction.java:21)
        at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:62)
        at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:159)
        at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:144)
        at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:400)
        at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:520)
        at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:124)
        at jdk.internal.reflect.GeneratedMethodAccessor2107.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:85)
        at com.google.common.eventbus.Subscriber.lambda$dispatchEvent$0(Subscriber.java:71)
        at com.vmware.vcf.common.tracing.TraceRunnable.run(TraceRunnable.java:59)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.util.concurrent.TimeoutException: null
        at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1960)
        at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2095)
        at com.vmware.vcf.inventory.sync.service.impl.VxRailManagerVersionSync.constructVersionDiffForVxManagers(VxRailManagerVersionSync.java:262)
        ... 21 common frames omitted

Environment

VMware Cloud Foundation 5.2.x

Cause

The default timeout value on SDDC Manager is 90 secs.

Resolution

Increase the SDDC Manager http client call wait time. 

Please Note: Multiple callers use this HttpClientService. Changing the timeout affects all the callers and can potentially cause an appearance of slowness before timing out.

1. SSH into the SDDC manager appliance with vcf and then elevate to root with su. 
   
   
2. Take a backup of the Operations Manager application-prod.properties file. 
   
   cp -p /etc/vmware/vcf/operationsmanager/application-prod.properties /home/vcf/application-prod.properties.OMbackup
   
   
3. Edit the Operations Manager application-prod.properties file.
    
   vi /etc/vmware/vcf/operationsmanager/application-prod.properties
   
   
4. Add new line - (180000 is doubling the default timeout to 3 minutes)
   
   http.client.timeout.milis=180000
   
   
5. Save and quit the file
   
   :wq or :x
    
6. Restart operationsmanager service 
   
   systemctl restart operationsmanager
   
   
7. Wait for the service to come up
   
   curl localhost/operationsmanager/about | json_pp

expected output:

{
   "description" : "Operations Manager",
   "id" : "ff6b221d-1e85-4c3f-b6d7-d1928a4c9cbe",
   "name" : "OPERATIONS_MANAGER",
   "serviceUrl" : "http://127.0.0.1/operationsmanager",
   "status" : "ACTIVE",
   "version" : "5.2.1-24397685"
}
   
   
8. Retry the SDDC workflow.