Host commission in SDDC Manager fails with error - Failed to validate vSAN HCL status

Products

VMware SDDC Manager VMware Cloud Foundation 5.x

Issue/Introduction

ESXi host commission in SDDC Manager fails with error Failed to validate vSAN HCL status

Error in /var/log/vmware/vcf/operationsmanager/operationsmanager.log

ERROR [vcf_om,6793fd6019650b807d52e05fbff94ecc,6c38] [c.v.v.h.f.v.s.impl.HostHclValidator,host-comm-decomm-exec-3] Failed to query HCL from LCM service.

org.springframework.web.client.HttpServerErrorException$ServiceUnavailable: 503 Service Temporarily Unavailable: "<html><EOL><EOL><head><title>503 Service Temporarily Unavailable</title></head><EOL><EOL><body><EOL><EOL><center><h1>503 Service Temporarily Unavailable</h1></center><EOL><EOL><hr><center>nginx</center><EOL><EOL></body><EOL><EOL></html><EOL><EOL>"

DEBUG [vcf_om,6793fd60d90e4a3d14865a6f7cdcac32,9e07] [c.v.e.s.c.c.v.vsan.VsanManagerBase,host-comm-decomm-exec-1] HCL check failed for test vSAN HCL DB up-to-date due to red status

2025-01-24T20:52:46.999+0000 ERROR [vcf_om,6793fd60d90e4a3d14865a6f7cdcac32,9e07] [c.v.v.h.c.s.i.CommissionHostsValidator,host-comm-decomm-exec-1] Host validation failed for Host: esxi01.example.com

DEBUG [vcf_om,6793fd5908ff951d25f8fc3623ae2893,d2f3] [c.v.v.h.c.s.i.CommissionHostsValidator,om-exec-17] esxi01.example.com: VSAN_HCL_VALIDATION_FAILURE

Environment

VMware Cloud Foundation 5.x

Cause

vSAN HCL DB on SDDC Manager is out of date.

Resolution

Take snapshot of SDDC Manager VM
SSH to sddc manager using vcf and su to root
Connect to lcm db using the below command
```
psql -h localhost -U postgres -d lcm
```
If SDDC Manager is 5.1.1 then use the below command
```
/usr/pgsql/13/bin/psql -U postgres -h localhost -d lcm
```
List contents on the table vsan_hcl_attribute
```
select * from vsan_hcl_attributes;
```
Take a note of local_data_timestamp value which records the latest HCL publish date for vsan_hcl_attributes
Note: if the value is outdated and/or not the latest then you get an error/warning as per the threshold value
Quit out of the lcm database by using
```
\q
```
Open the below link in your HTML browser:
1. https://vvs.broadcom.com/service/vsan/all.json (Right click, Save as)
2. Save the file as all.json on your local machine. If you are unable to save the file, you must copy the entire content and create a new file with extension".json"
Copy the latest all.json file from your local machine to /home/vcf folder onto your sddc manager via tools like WinSCP

Use the below command to grab the access token

curl --location 'http://<sddcmanager_fqdn>/v1/tokens' --header 'Content-Type: application/json' --data-raw '{"username":"<SSO_USERNAME>","password":"<SSO_PASSWORD>"}'

cd to /home/vcf (where the latest local copy of all.json is placed)
Upload the HCL file using the HCL upload API using the below command
```
curl -X PUT --location 'http://<sddcmanager_fqdn>/v1/vsan-hcl/content' --header 'Authorization: Bearer <put the token grabbed from step 8>' --header 'Content-Type: text/plain' [email protected]
```
If Step 10 fails with error: {"errorCode": "Unauthorized", "message" JWT strings must contain exactly 2 period characters. Found : 0"}, refer Additional Information section below.
Post successful execution of the API the contents of local_data_timestamp in vsan_hcl_attributes at lcm db will get updated to a new value, please verify the same as stated in step # 4
Retry host commission from SDDC Manager

Additional Information

If Step 10 fails with error: {"errorCode": "Unauthorized", "message" JWT strings must contain exactly 2 period characters. Found : 0"}, make following changes in step 8 and 10.

In Step 8, use the below command to grab the access token

curl -k --location 'https://<sddcmanager_fqdn>/v1/tokens' --header 'Content-Type: application/json' --data-raw '{"username":"<SSO_USERNAME>","password":"<SSO_PASSWORD>"}'
Example: Access Token JSON Response
When you request a token from SDDC Manager, you get a JSON response like this:

{ "accessToken": "<your_access_token_here>", "refreshToken": { "id": "<your_refresh_token_here>" } }

In Step 10, upload the HCL file using the HCL upload API using the below command

curl -k -X PUT --location 'https://<sddcmanager_fqdn>/v1/vsan-hcl/content' --header "Authorization: Bearer <your_access_token_here from step 8>" --header 'Content-Type: text/plain' [email protected]