ESXi upgrade failing with "ESX Upgrade and interop compatibility checks failed." on the SDDC UI

Products

VMware SDDC Manager VMware Cloud Foundation 5.x

Issue/Introduction

ESXi Upgrade fails with compatibility error - "ESX Upgrade and interop compatibility checks failed."

lcm-debug.log - /var/log/vmware/vcf/lcm
2024-08-30T14:15:03.155+0000 ERROR [vcf_lcm,0000000000000000,0000,upgradeId=6d91d60c-2fa1-4696-9d0a-a0d8f5b64807,resourceType=ESX_CLUSTER,resourceId=afdeb7e2-21c8-4f1c-b2ac-aa2c7cc8768b,bundleElementId=55f34f18-c075-4875-b76e-1b44c8e35af0] [c.v.e.s.l.p.i.c.VlcmPrimitiveImpl,Upgrade-1] Failed to calculate ESX upgrade compatibility for cluster cluster_name
java.lang.NullPointerException: Cannot invoke "com.vmware.evo.sddc.common.version.VMwareProductVersion.hashCode()" because "this.version" is null
at com.vmware.vcf.compatibility.translators.VcfProductVersion.hashCode(VcfProductVersion.java:49)
at java.base/java.util.HashMap.hash(HashMap.java:338)
at java.base/java.util.HashMap.put(HashMap.java:610)
at java.base/java.util.HashSet.add(HashSet.java:221)
at com.vmware.vcf.compatibility.validator.internal.CompatibilityVersionUtil.lambda$getVcfProductVersionSet$2(CompatibilityVersionUtil.java:80)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
2024-08-30T14:15:03.170+0000 DEBUG [vcf_lcm,b21798035cf4476e,e565] [c.v.e.s.l.a.i.i.LogicalInventoryClient,http-nio-127.0.0.1-7400-exec-6] logical inventory - get domain inventory http://localhost/inventory/domains/3a5c9418-2b44-4253-8576-824460d1a3e3/inventory
2024-08-30T14:15:03.197+0000 DEBUG [vcf_lcm,0000000000000000,0000,upgradeId=6d91d60c-2fa1-4696-9d0a-a0d8f5b64807,resourceType=ESX_CLUSTER,resourceId=afdeb7e2-21c8-4f1c-b2ac-aa2c7cc8768b,bundleElementId=55f34f18-c075-4875-b76e-1b44c8e35af0] [c.v.e.s.l.p.i.c.VlcmPrimitiveImpl,Upgrade-1] Error: {"errorType":"RECOVERABLE","stage":"START","errorCode":"com.vmware.vcf.error.runtime.esx.vvs.upgrade.compatibility.failed","errorDescription":"ESX Upgrade and interop compatibility checks failed.","metadata":"ESX upgrade bundle is not compatible with your existing version. Refer to the https://kb.vmware.com/s/article/91728 to address incompatibilities and proceed with the upgrade. \nPlease verify product versions stored in VCF inventory are in-sync with actual product version. \nPlease use SDDC Manager Compatibility APIs to verify whether the ESXI target version is interoperabale with other BOM components and it can be upgraded from current version to target version. \nFor more information on the not compatible product pairs, check the activity log.\nCheck for errors in the lcm log files located on SDDC Manager under /var/log/vmware/vcf/lcm.","metadataCodes":["ESX_VVS_INCOMPATIBLE_UPGRADE","ESX_VVS_INCOMPATIBLE_UPGRADE","LCM_LOG"],"metadataAttributes":{"LCM_LOG_LOCATION":"/var/log/vmware/vcf/lcm","LCM_HOST_ADDRESS":"127.0.0.1"},"referenceToken":"1SL0OS"}
2024-08-30T14:15:03.198+0000 ERROR [vcf_lcm,0000000000000000,0000,upgradeId=6d91d60c-2fa1-4696-9d0a-a0d8f5b64807,resourceType=ESX_CLUSTER,resourceId=afdeb7e2-21c8-4f1c-b2ac-aa2c7cc8768b,bundleElementId=55f34f18-c075-4875-b76e-1b44c8e35af0] [c.v.e.s.l.p.i.c.VlcmPrimitiveImpl,Upgrade-1] UpgradeError while calculating VVS compatibility, stage: START, errorType: RECOVERABLE,errorCode: com.vmware.vcf.error.runtime.esx.vvs.upgrade.compatibility.failed, errorDescription: ESX Upgrade and interop compatibility checks failed., metadata ESX upgrade bundle is not compatible with your existing version. Refer to the https://kb.vmware.com/s/article/91728 to address incompatibilities and proceed with the upgrade.

Environment

VMware Cloud Foundation 5.2.x

Cause

The issue is encountered due to a case mismatch of the ESXi hostname in SDDC Manager and vCenter server inventory.

Steps to identify the hostname from SDDC DB -

1. Connect to SDDC DB -

 psql -h localhost -U postgres

2. Change to Platform DB -

\c platform

3. Check the host id from the Host Table -

select id, hostname from host;

Resolution

NOTE - Make sure to take a snapshot of the SDDC manager before following the steps below -

1. Take a snapshot of the SDDC Manager
2. Take an SSH to SDDC login as vcf and elevate to root.
3. Run the following commands to update the hostname in the SDDC DB.
  1. Connect to SDDC DB -
```
 psql -h localhost -U postgres 
```
  2. Change to Platform DB -
```
\c platform
```
  3. Update the hostname of the hosts as per the vCenter inventory -
```
update host set hostname='xxx' where hostname='xxxxx';
```