vCenter Update in a VMware Cloud Foundation environment Stuck on VSAN HCL Update
Article ID: 382408
Updated On:
Products
Issue/Introduction
During the update of vCenter in a VCF environment the upgrade process takes an extended amount of time in the final step:
VCENTER UPGRADE POST INSTALL VSAN HCL UPDATE
We see the following log entries in lcm-debug.log.
/var/log/vmware/vcf/lcm/lcm-debug.log:
2024-10-30T20:32:15.415+0000 DEBUG [vcf_lcm,0000000000000000,0000,upgradeId=########-####-####-####-############,resourceType=VCENTER,resourceId=########-####-####-####-############,bundleElementId=9708626a-175b-49af-8eea-3c7ec6ad28f4] [c.v.e.s.l.p.i.vsan.VsanHealthService,Upgrade-1] Getting Vsan connection for https://vcenter.example.com/sdk2024-10-30T20:32:16.411+0000 DEBUG [vcf_lcm,0000000000000000,0000,upgradeId=########-####-####-####-############,resourceType=VCENTER,resourceId=########-####-####-####-############,bundleElementId=9708626a-175b-49af-8eea-3c7ec6ad28f4] [c.v.e.s.l.p.c.c.VsanHealthConnection,Upgrade-1] Connecting to Virtual SAN Health Service...2024-10-30T20:32:17.087+0000 INFO [vcf_lcm,0000000000000000,0000,upgradeId=########-####-####-####-############,resourceType=VCENTER,resourceId=########-####-####-####-############,bundleElementId=9708626a-175b-49af-8eea-3c7ec6ad28f4] [c.v.e.s.l.p.c.c.VsanHealthConnection,Upgrade-1] Virtual SAN Health Connection created...2024-10-30T20:32:17.087+0000 INFO [vcf_lcm,0000000000000000,0000,upgradeId=########-####-####-####-############,resourceType=VCENTER,resourceId=########-####-####-####-############,bundleElementId=9708626a-175b-49af-8eea-3c7ec6ad28f4] [c.v.e.s.l.p.i.vsan.VsanHealthService,Upgrade-1] Got Vsan connection for https://vcenter.example.com/sdk2024-10-30T20:32:17.388+0000 DEBUG [vcf_lcm,0000000000000000,0000,upgradeId=########-####-####-####-############,resourceType=VCENTER,resourceId=########-####-####-####-############,bundleElementId=9708626a-175b-49af-8eea-3c7ec6ad28f4] [c.v.e.s.l.p.i.vsan.VsanHealthService,Upgrade-1] Current HCL last udpated date in seconds from epoch 1727073120 i.e. 2024-09-23T08:32:00+02:002024-10-30T20:32:17.388+0000 WARN [vcf_lcm,0000000000000000,0000,upgradeId=########-####-####-####-############,resourceType=VCENTER,resourceId=########-####-####-####-############,bundleElementId=9708626a-175b-49af-8eea-3c7ec6ad28f4] [c.v.e.s.l.p.i.vsan.VsanHealthService,Upgrade-1] Expected HCL timestamp 1727080320 is NOT same as actual HCL timestamp 17270731202024-10-30T20:32:17.389+0000 ERROR [vcf_lcm,0000000000000000,0000,upgradeId=########-####-####-####-############,resourceType=VCENTER,resourceId=########-####-####-####-############,bundleElementId=9708626a-175b-49af-8eea-3c7ec6ad28f4] [c.v.e.s.l.p.i.vsan.VsanHealthService,Upgrade-1] Failed to upload the VSAN HCL DB.2024-10-30T20:32:17.389+0000 ERROR [vcf_lcm,0000000000000000,0000,upgradeId=########-####-####-####-############,resourceType=VCENTER,resourceId=########-####-####-####-############,bundleElementId=9708626a-175b-49af-8eea-3c7ec6ad28f4] [c.v.e.s.l.p.i.vcenter.VCenterUpdater,Upgrade-1] Fatal failure while performing VSAN HCL update2024-10-30T20:32:17.390+0000 ERROR [vcf_lcm,0000000000000000,0000,upgradeId=########-####-####-####-############,resourceType=VCENTER,resourceId=########-####-####-####-############,bundleElementId=9708626a-175b-49af-8eea-3c7ec6ad28f4] [c.v.e.s.l.p.i.vcenter.VCenterUpdater,Upgrade-1] VSAN HCL Update failed at attempt number 40, sleeping...
Environment
VMware Cloud Foundation 4.5.x
VMware Cloud Foundation 5.x
Cause
This issue is related to a discrepancy between the expected and actual timestamps of the VSAN HCL due to time zone differences.
When the vCenter is configured in a non-UTC time zone, it encounters an incorrect timestamp comparison during the VSAN HCL update, leading to prolonged processing times and eventual failure.
Resolution
Update the timezone in the vCenter to UTC using the vCenter VAMI page:
https://<vcenter_fqdn>:5480
Detailed steps are available here:
Configure the System Time Zone and Time Synchronization Settings
Additional Information
You may observe the following error within the /var/log/vmware/vcf/lcm/lcm-debug.log even after updating the time zone to UTC and the error may persist:
Post install steps for VCENTER <vcsa hostname> failed. VSAN HCL Update failed due to exception Failed to upload the VSAN HCL DB. VSAN HCL DB timestamp is not updated in vCenter even after successful API.If it happens to be an offline environment or the vSAN HCL is failing to update you might be able to rectify this error by updating the vSAN HCL Manually by following this KB:
Download/Upload HCL data for VCF
If the upgrade is still being processed, you may not have to intervene. If it's in a failed state, retry the vCenter upgrade from SDDC and observe any errors in the LCM.log if applicable or observe if the upgrade has completed successfully.
Feedback
