Syncing vSAN HCL to vCenter is failing with the error : "Failed to upload the VSAN HCL DB. VSAN HCL DB timestamp is not updated in vCenter even after successful API"

Products

• VMware SDDC Manager

Issue/Introduction

When the json file is updated https://vvs.broadcom.com/service/vsan/all.json & the sync between the between the vSAN HCL of the vCenter & SDDC happens it is failing with the error : "Failed to upload the VSAN HCL DB. VSAN HCL DB timestamp is not updated in vCenter even after successful API" with no impact to the pre-checks on the SDDC manager.

When the sync is attempted we see the below logs :

/var/log/vmware/vcf/lcm/lcm-debug.log

yyyy-mm-ddT11:43:54.870+0800 INFO  [vcf_lcm,0000000000000000,0000] [c.v.e.s.l.p.i.vsan.VsanHealthService,HCL-Sync-1] For the new HCL, the timestamp is 1763007154 & jsonUpdatedTime is 1763007120
yyyy-mm-ddT11:43:54.871+0800 DEBUG [vcf_lcm,0000000000000000,0000] [c.v.e.s.l.p.i.vsan.VsanHealthService,HCL-Sync-1] We found entry for jsonUpdatedTime, hence using jsonUpdatedTime as timestamp value for new HCL
yyyy-mm-ddT11:43:58.883+0800 WARN  [vcf_lcm,0000000000000000,0000] [c.v.e.s.l.p.i.vsan.VsanHealthService,HCL-Sync-1] Expected HCL timestamp 1763007120 is NOT same as actual HCL timestamp 1762978320
yyyy-mm-ddT11:43:59.178+0800 INFO  [vcf_lcm,0000000000000000,0000] [c.v.e.s.l.p.i.vsan.VsanHealthService,HCL-Sync-1] For the new HCL, the timestamp is 1763007154 & jsonUpdatedTime is 1763007120
yyyy-mm-ddT11:43:59.178+0800 DEBUG [vcf_lcm,0000000000000000,0000] [c.v.e.s.l.p.i.vsan.VsanHealthService,HCL-Sync-1] We found entry for jsonUpdatedTime, hence using jsonUpdatedTime as timestamp value for new HCL
yyyy-mm-ddT11:44:02.646+0800 WARN  [vcf_lcm,0000000000000000,0000] [c.v.e.s.l.p.i.vsan.VsanHealthService,HCL-Sync-1] Expected HCL timestamp 1763007120 is NOT same as actual HCL timestamp 1762978320
yyyy-mm-ddT11:44:02.953+0800 INFO  [vcf_lcm,0000000000000000,0000] [c.v.e.s.l.p.i.vsan.VsanHealthService,HCL-Sync-1] For the new HCL, the timestamp is 1763007154 & jsonUpdatedTime is 1763007120
yyyy-mm-ddT11:44:02.953+0800 DEBUG [vcf_lcm,0000000000000000,0000] [c.v.e.s.l.p.i.vsan.VsanHealthService,HCL-Sync-1] We found entry for jsonUpdatedTime, hence using jsonUpdatedTime as timestamp value for new HCL
yyyy-mm-ddT11:44:06.563+0800 WARN  [vcf_lcm,0000000000000000,0000] [c.v.e.s.l.p.i.vsan.VsanHealthService,HCL-Sync-1] Expected HCL timestamp 1763007120 is NOT same as actual HCL timestamp 1762978320
yyyy-mm-ddT11:44:06.585+0800 ERROR [vcf_lcm,0000000000000000,0000] [c.v.e.s.l.s.impl.VsanHclSyncRunner,HCL-Sync-1] Failed to update vSAN HCL in vCenter: <hostname> and cluster <cluster_name> with exception Failed to upload the VSAN HCL DB. VSAN HCL DB timestamp is not updated in vCenter even after successful API.
yyyy-mm-ddT11:44:06.585+0800 ERROR [vcf_lcm,0000000000000000,0000] [c.v.e.s.l.s.impl.VsanHclSyncRunner,HCL-Sync-1] Uploading vSAN HCL to vCenter: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx failed with exception: Failed to upload the VSAN HCL DB. VSAN HCL DB timestamp is not updated in vCenter even after successful API.
com.vmware.evo.sddc.lcm.model.error.LcmException: Failed to upload the VSAN HCL DB. VSAN HCL DB timestamp is not updated in vCenter even after successful API.

 

Environment

VMware Cloud Foundation 4.5.x

VMware Cloud Foundation 5.x

Cause

vCenter & SDDC manager both are in a different timezone other than UTC due to which it encounters an incorrect timestamp comparison during the VSAN HCL update, leading to prolonged processing times and eventual failure.

Resolution

1. Update the timezone to UTC in the vCenter using the KB : Setting the Time Zone in the vCenter Server Appliance

2. Update the timezone to UTC in SDDC. 

2. Reboot both SDDC & vCenter.