vCenter Upgrade from SDDC manager fails with error "Failed to upload the VSAN HCL DB. VSAN HCL DB timestamp is not updated in vCenter even after successful API"

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

Symptoms:

 

• Running vCenter prechecks from the SDDC manager UI prior to upgrade does not report any vSAN HCL out of date issues or has already been cleared using KB article 316041
  
  
• Attempt to upgrade fails and reports the following error on the lcm logs:
  
  /var/log/vmware/vcf/lcm/lcm-debug.log
  ERROR [vcf_lcm,0000000000000000,0000,upgradeId=abcdxxxxxx-abcd-abcd-abcd-abcdxxxxxxxx,resourceType=VCENTER,
resourceId=abcdxxxxxx-abcd-abcd-abcd-abcdxxxxxxxx,d,bundleElementId=abcdxxxxxx-abcd-abcd-abcd-abcdxxxxxxxx] 
[c.v.e.s.l.p.i.vcenter.VCenterCommon,Upgrade-2] Fatal failure while performing VSAN HCL update
com.vmware.evo.sddc.lcm.model.error.LcmException: 
Failed to upload the VSAN HCL DB. VSAN HCL DB timestamp is not updated in vCenter even after successful API.

Environment

VMware Cloud Foundation 5.x

Resolution

Important Note: This procedure should not be used directly if the upgrade pre-check test fails with the error: "vSAN HCL DB on SDDC Manager is out of date".

In that situation refer first to article 316041 to make sure the SDDC manager is up to date and pre-check tests are clear.

 

1. SSH to the SDDC Manager using vcf and su to root
   
   
2. Run the following query to the database and retrieve the data_timestamp values:
   

   # /usr/pgsql/13/bin/psql -U postgres -h localhost -d lcm -c 'select * from vsan_hcl_attributes;'

3. Visit any online Epoch timestamp conversion tool and convert the timestamp taken from step 2 to a human-readable date.
   
   for example "1721122728" should be "July 16, 2024, 2:38 AM PDT"
   
   Take note of both values as they are going to be used later.
   
   
4. Open the below link in HTML browser with internet access: 
   https://vvs.broadcom.com/service/vsan/all.json  (Right click, Save as)
   
   
5. Save the file with a ".json" extension
   example given: updated-hcl-all.json
   
   
6. Edit the .json file and make sure that the "timestamp" and "jsonUpdatedTime" values equals or are more recent than the values retrieved from step 3.
   
   
7. Login to the vSphere Client and upload the file to the vCenter:
   
   
   • vCenter > Configure > vSAN > Update > Update from file
     or
   • Cluster > Monitor > vSAN > Health/Skyline Health > Hardware Compatibility > vSAN HCL DB up-to-date > Update from file
     
8. Verify that the vCenter date is updated.
   
   
9. Retry the vCenter upgrade operation from SDDC manager.

Additional Information

Updating the vSAN HCL database manually
SDDC Manager upgrade precheck error : vSAN HCL DB on SDDC Manager is out of date