Host commission in SDDC Manager fails with error - Failed to validate vSAN HCL status
search cancel

Host commission in SDDC Manager fails with error - Failed to validate vSAN HCL status

book

Article ID: 387124

calendar_today

Updated On:

Products

VMware SDDC Manager VMware Cloud Foundation 5.x

Issue/Introduction

  • ESXi host commission in SDDC Manager fails with error Failed to validate vSAN HCL status
  • Error in /var/log/vmware/vcf/operationsmanager/operationsmanager.log 
    ERROR [vcf_om,6793fd6019650b807d52e05fbff94ecc,6c38] [c.v.v.h.f.v.s.impl.HostHclValidator,host-comm-decomm-exec-3] Failed to query HCL from LCM service.

org.springframework.web.client.HttpServerErrorException$ServiceUnavailable: 503 Service Temporarily Unavailable: "<html><EOL><EOL><head><title>503 Service Temporarily Unavailable</title></head><EOL><EOL><body><EOL><EOL><center><h1>503 Service Temporarily Unavailable</h1></center><EOL><EOL><hr><center>nginx</center><EOL><EOL></body><EOL><EOL></html><EOL><EOL>"

    DEBUG [vcf_om,6793fd60d90e4a3d14865a6f7cdcac32,9e07] [c.v.e.s.c.c.v.vsan.VsanManagerBase,host-comm-decomm-exec-1] HCL check failed for test vSAN HCL DB up-to-date due to red status

2025-01-24T20:52:46.999+0000 ERROR [vcf_om,6793fd60d90e4a3d14865a6f7cdcac32,9e07] [c.v.v.h.c.s.i.CommissionHostsValidator,host-comm-decomm-exec-1] Host validation failed for Host: esxi01.example.com

    DEBUG [vcf_om,6793fd5908ff951d25f8fc3623ae2893,d2f3] [c.v.v.h.c.s.i.CommissionHostsValidator,om-exec-17] esxi01.example.com: VSAN_HCL_VALIDATION_FAILURE

Environment

VMware Cloud Foundation 5.x

Cause

vsan hcl db in SDDC Manager is out of date

Resolution

  1. Take snapshot of SDDC Manager VM
  2. SSH to sddc manager using vcf and su to root
  3. Connect to lcm db using the below command 
    psql -h localhost -U postgres -d lcm
    If SDDC Manager is 5.1.1 then use the below command 
    /usr/pgsql/13/bin/psql -U postgres -h localhost -d lcm
  4. List contents on the table vsan_hcl_attribute 
    select * from vsan_hcl_attributes;

    Take a note of local_data_timestamp value which records the latest HCL publish date for vsan_hcl_attributes
    Note: if the value is outdated and/or not the latest then you get an error/warning as per the threshold value

  5. Quit out of the lcm database by using 
    \q
  6. Open the below link in your HTML browser:
    1. https://partnerweb.vmware.com/service/vsan/all.json (Right click, Save as)
    2. Save the file as all.json on your local machine. If you are unable to save the file, you must copy the entire content and create a new file with extension".json"
  7. Copy the latest all.json file from your local machine to /home/vcf folder onto your sddc manager via tools like WinSCP
  8. Use the below command to grab the access token 
    curl --location 'http://<sddcmanager_fqdn>/v1/tokens' --header 'Content-Type: application/json' --data-raw '{"username":"<SSO_USERNAME>","password":"<SSO_PASSWORD>"}'
  9. cd to /home/vcf (where the latest local copy of all.json is placed)
  10. Upload the HCL file using the HCL upload API using the below command 
    curl -X PUT --location 'http://<sddcmanager_fqdn>/v1/vsan-hcl/content' --header 'Authorization: Bearer <put the token grabbed from step 8>' --header 'Content-Type: text/plain' [email protected]
  11. Post successful execution of the API the contents of local_data_timestamp in vsan_hcl_attributes at lcm db will get updated to a new value, please verify the same as stated in step # 4
  12. Retry host commission from SDDC Manager
Powered by Wolken Software