SDDC Manager Displays Incorrect ESXi Host Version After Out-of-Band Patching or Manual Upgrade

Products

• VMware Cloud Foundation

Issue/Introduction

When performing out-of-band (OOB) upgrades on ESXi hosts outside of the Lifecycle Management workflow (e.g., directly via vSphere Lifecycle Manager/vLCM), the SDDC Manager inventory database may fail to synchronize, causing the user interface to display older build versions. Attempting to resolve this using API calls (curl -X PATCH) directly in Bash may fail with VCF_RUNTIME_ERROR or URL syntax errors if authentication headers are missing or if multi-line inputs fail to parse properly.

Environment

• VMware Cloud Foundation 4.x
• VMware Cloud Foundation 5.x

Cause

SDDC Manager maintains its own inventory database of host versions. When an ESXi host is upgraded outside of the SDDC Manager lifecycle management workflow (out-of-band upgrade), the inventory database is not automatically updated. This causes the SDDC Manager UI to display the old version even though the host is running the new version. While API methods exist, direct PostgreSQL database updates provide the cleanest, most reliable workaround when API calls encounter runtime or formatting exceptions.

Resolution

Before making any changes, make sure to take a snapshot of the SDDC VM Taking and restoring snapshots of the SDDC Manager appliance

To resolve the issue, manual synchronization of the SDDC Manager inventory database.

Workaround 1

1. Verify Host State: SSH to the affected ESXi host as a root user and run vmware -v to confirm the actual build version.
2. Access SDDC Manager: SSH to the SDDC Manager appliance as a vcf user and switch to root user
3. Identify Host: Retrieve the host ID (UUID) for the affected host(s): curl -s localhost/inventory/hosts | jq -r '.[]|{id,hostName}'
4. SDDC Database Check: Access the PostgreSQL platform database: /usr/pgsql/13/bin/psql -U postgres -h localhost -d platform
   1. Identify the specific host record: SELECT id, hostname, version FROM host;
   2. Exit out of the PostgreSQL using \q
5. Execute Update: Update the host version: curl -X PATCH 'localhost/inventory/entities/<host-id>' \ -d '{"version":"<version>", "type":"ESXI"}' \ -H 'Content-Type:application/json'
   (Replace <version> with the correct ESXi version retrieved from Step 1 - it will be in the format of eg: 8.0.3-####### and <host-id> should be replaced with the UUID retried from step 3)
6. Restart Services: Restart the LCM services to clear the cache: /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh
7. Verify: Refresh the SDDC Manager UI and confirm the correct version is displayed.

Workaround 2

• Access the PostgreSQL Shell Open an SSH session to the SDDC Manager appliance, switch to the root user, and connect to the local platform database:

  /usr/pgsql/13/bin/psql -U postgres -h localhost -d platform
  

• Execute Batch Host Version Update At the platform=# prompt, run a SQL UPDATE statement to update all affected ESXi host records simultaneously using your target ESXi build number and host FQDNs:

  UPDATE host SET version='<target-build-number>' WHERE hostname IN ('<esx-host-fqdn-01>', '<esx-host-fqdn-02>', '<esx-host-fqdn-03>', '<esx-host-fqdn-04>');

• Verify Database Records and Exit Confirm that the database updated the rows properly before exiting:

  SELECT hostname, version FROM host WHERE hostname LIKE 'esx%';
  

  Exit the PostgreSQL shell:

  \q
  

• Confirm UI Inventory Refresh

  • Log into the SDDC Manager UI (https://).

  • Navigate to Inventory > Hosts.

  • Verify that all hosts display the newly installed software version (e.g., 8.0.3-25595708).

 

Additional Information

Note: To avoid future version discrepancies, prioritize the use of SDDC Manager Lifecycle Management for all ESXi host upgrades.

Synchronize inventory versions after out-of-band upgrade in a VMware Cloud Foundation Environment

SDDC/VCF environments and Out Of Band Changes