Generating the hardware compatibility report via the vCenter GUI is falling when a Proxy is in use.
search cancel

Generating the hardware compatibility report via the vCenter GUI is falling when a Proxy is in use.

book

Article ID: 438438

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

When generating the hardware compatibility report via the vCenter GUI you receive the following messages [either/ or]
"There is compatibility data update in progress. Wait for the compatibility data update to finish and retry the operation."
or
"Check vCenter Server Appliance internet connection to the VMware Verification Service at 'vvs.esp.vmware.com' and 'auth.esp.vmware.com'

vSphere online environment with Proxy in use, download token in place. 
URLs are whitelisted as per https://knowledge.broadcom.com/external/article/327186/public-url-list-for-sddc-manager.html 

From the vCenter logs we see that the <model name> is not present, i.e. no VCG data is downloaded.
Subsequently, the vCenter will try to go online via VVS to find that particular model. 

# less -i var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log
<date && time> info vmware-vum-server[32959] [Originator@6876 sub=HostInfo] [HostInfo 1103] Invoking hcl-constraints with hostId: host-46, targetVersion: ESXi 9.X (timeout = 300)

# hcl_python_lib.log
<date && time> INFO compatibility.server Getting server compatibility for <model name> for version ESXi 9.x
<date && time> INFO compatibility.server Found servers for model <model name>: None    
<date && time> INFO compatibility.server Servers found in local datastore: None
<date && time> INFO compatibility.server Searching compatibility info online for model: <model name>
<date && time> INFO compatibility.vvs.rest.configuration Determining proxy for target host: 'localhost' (scheme = http).

HCL will most likely NOT find the server info from its local store, see NONE
<date && time> INFO compatibility.vvs.rest.configuration HTTP proxy for host 'localhost' is 'None'.
<date && time> INFO compatibility.vvs.rest.vvs_search Searching for server; smbiosmodel = <model name>...

so it'll try to search for it online via VVS, to do that it will try to retrieve a token but failed to do so.
<date && time> INFO compatibility.vvs.rest.vvs_client Requesting a new JWT token for authentication from 'http://localhost:1080/external-os/http1/auth.esp.vmware.com/443/api/auth/v1/tokens'.

Envoy will answer with "no_healthy_upstream" flag in envoy log, initially indicating VVS service was not running correctly, thus triggering the error
# cd /var/log/vmware/envoy/
envoy-access.log <date && time> info envoy[2852] [Originator@6876 sub=Default] 2026-03-24T12:53:13.412Z POST /external-os/http1/auth.esp.vmware.com/443/api/auth/v1/tokens 503 no_healthy_upstream UH 0 19 - 240076 - - 127.0.0.1:54728 HTTP/1.1 - 127.0.0.1:1080  0 0 - - - [<IPV6 IP reference>]:443 - - -

# grep -Ri no_healthy_upstream | awk '{print $1, $8}' | sort | grep -i auth.esp.vmware.com
envoy-access-##.log:<date && time> /external-os/http1/auth.esp.vmware.com/443/api/auth/v1/tokens
envoy-access-##.log:<date && time> /external-os/http1/auth.esp.vmware.com/443/api/auth/v1/tokens
envoy-access-##.log:<date && time> /external-os/http1/auth.esp.vmware.com/443/api/auth/v1/tokens
envoy-access-##.log:<date && time> /external-os/http1/auth.esp.vmware.com/443/api/auth/v1/tokens
envoy-access-##.log:<date && time> /external-os/http1/auth.esp.vmware.com/443/api/auth/v1/tokens


All other testing works fine, download tokens are in place, sync updates works, curl also works

# curl -kvvv https://vvs.esp.vmware.com  
[...]
CONNECT vvs.esp.vmware.com:443 HTTP/1.1
Host: vvs.esp.vmware.com:443
User-Agent: curl/8.7.1
Proxy-Connection: Keep-Alive
< HTTP/1.1 200 OK
[...]

same for 
curl -kvvv https://vvs.esp.vmware.com
curl -kvvv https://vvs.broadcom.com
curl -kvvv https://auth.esp.vmware.com

Environment

vSphere 9.X

Cause

There is an issue, when using a Proxy for the vCenter, in an online environment.
We can't download the VMware Compatibility Guide - i.e. VCG, but we can update it via sync updates.  

Resolution

VMware by Broadcom Engineering is aware about this behavior, and is planning to address the issue in a future release.

As a workaround please use the seps described in
https://knowledge.broadcom.com/external/article/405839/steps-to-upgrade-manually-the-vcg-databa.html 

Steps to follow [amended for ONLINE vCenter]
1. Login to the vCenter using the root account and access the bash shell.

2. Run the following command:
# grep 'client_' /usr/lib/vmware-updatemgr/config/vvs-config.json

3. Note the client_id and client_secret values. These are going to be used in later steps
Download a copy of the VCG database via curl [line by line or complete block]

CLIENT_ID="<replace with values from step 2>"
CLIENT_SECRET="<replace with values from step 2>"
JSON=$(curl -d "client_id=${CLIENT_ID}&client_secret=${CLIENT_SECRET}&grant_type=client_credentials" -X POST https://auth.esp.vmware.com/api/auth/v1/tokens)
TOKEN=$(echo $JSON | jq -r .access_token)
curl -L -H "X-Vmw-Esp-Client: $TOKEN" -X GET "https://vvs.esp.vmware.com/v1/compatible/vcg/bundles/all?format=gz" -o vlcm-vcg-offline.gz

4. Unzip the file:
# gzip -d vlcm-vcg-offline.gz

5. Update the local vLCM VCG database from our offline file
# /usr/lib/vmware-updatemgr/python/hcl/hcl_datastore.py update-offline --filePath /<path to the file>/vlcm-vcg-offline

6. Then to check:
# /usr/lib/vmware-updatemgr/python/hcl/hcl_datastore.py information


Once the VCG is downloaded, the updates will follow via 
Lifecycle Manager > Actions > Updates > Sync Updates

Sync can and will refresh the VCG data that is stored on VCenter.

Powered by Wolken Software