Upgrade of ESXi host managed by SDDC fails with an error “The Host is incompatible with the Patch/upgrade iso file”

Products

• VMware Cloud Foundation
• VMware vSphere ESXi

Issue/Introduction

Message: The host "<UUID>" is incompatible with the Patch/upgrade iso file
Remediation Message: High: Incompatible patch or upgrade files. Please verify that the patch file is compatible for host. Refer LCM and VUM log file. Check for errors in the lcm log files located on SDDC Manager under /var/log/vmware/vcf/lcm. Please retry the upgrade once the upgrade is available again.

The issue can be seen in both VUM ISO upgrade and VLCM cluster that is doing an upgrade on base image

In /var/log/vmware/vcf/lcm/lcm-debug.log, you read log entries similar to:

YYYY-MM-DDTHH:MIN:SEC+0000 DEBUG [vcf_lcm,b7d5d5b2cffa4f15,dd95] [c.v.e.s.l.s.i.UpgradeGetterService,http-nio-127.0.0.1-7400-exec-2] Getting normalized upgrade status for parallel upgrades
YYYY-MM-DDTHH:MIN:SEC+0000 DEBUG [vcf_lcm,b7d5d5b2cffa4f15,dd95] [c.v.e.s.l.s.i.UpgradeGetterService,http-nio-127.0.0.1-7400-exec-2] Upgrade status COMPLETED_WITH_FAILURE for child upgrade: e45cdf4b-af38-4942-83dc-1adf3710b0e9
YYYY-MM-DDTHH:MIN:SEC+0000 DEBUG [vcf_lcm,b7d5d5b2cffa4f15,dd95] [c.v.e.s.l.s.i.UpgradeGetterService,http-nio-127.0.0.1-7400-exec-2] Normalized Parent upgrade status COMPLETED_WITH_FAILURE for parent id: 740c1bfe-1dc8-45fa-8f8e-070f8f0071b6
YYYY-MM-DDTHH:MIN:SEC+0000 DEBUG [vcf_lcm,b7d5d5b2cffa4f15,dd95] [c.v.e.s.l.s.i.UpgradeGetterService,http-nio-127.0.0.1-7400-exec-2] Building normalized user visible upgrade for upgrade id: 740c1bfe-1dc8-45fa-8f8e-070f8f0071b6
YYYY-MM-DDTHH:MIN:SEC+0000 DEBUG [vcf_lcm,b7d5d5b2cffa4f15,dd95] [c.v.e.s.l.s.i.UpgradeGetterService,http-nio-127.0.0.1-7400-exec-2] Built normalized user visible upgrade for upgrade id: 740c1bfe-1dc8-45fa-8f8e-070f8f0071b6
YYYY-MM-DDTHH:MIN:SEC+0000 DEBUG [vcf_lcm,b7d5d5b2cffa4f15,dd95] [c.v.e.s.l.s.i.UpgradeGetterService,http-nio-127.0.0.1-7400-exec-2] Getting normalized parent upgrade for upgrade id: 19fe5049-7210-45a3-9b69-a9b11688b28a
YYYY-MM-DDTHH:MIN:SEC+0000 DEBUG [vcf_lcm,b7d5d5b2cffa4f15,dd95] [c.v.e.s.l.s.i.UpgradeGetterService,http-nio-127.0.0.1-7400-exec-2] Getting normalized upgrade status for parallel upgrades
YYYY-MM-DDTHH:MIN:SEC+0000 DEBUG [vcf_lcm,b7d5d5b2cffa4f15,dd95] [c.v.e.s.l.s.i.UpgradeGetterService,http-nio-127.0.0.1-7400-exec-2] Upgrade status COMPLETED_WITH_FAILURE for child upgrade: 4bf2fd65-4995-4b9c-b429-e93cd43e58c4
YYYY-MM-DDTHH:MIN:SEC+0000 ERROR [vcf_lcm,66e1f966f308764807581e930f06328e,7fac] [c.v.evo.sddc.lcm.model.task.SubTask,http-nio-127.0.0.1-7400-exec-1] Upgrade error occured: The host 4b8441b3-66c5-4862-81a5-863d49f062d5 is incompatible with the Patch/upgrade iso file

In ESXi host navigate to /var/log/vua.log to find entries related to Dirty bit in vFAT partition:

-->     <test>
-->       <name>VFAT_CORRUPTION</name>
-->       <expected>
-->         <value>True</value>
-->       </expected>
-->       <found>
-->         <value>False</value>
-->       </found>
-->       <result>ERROR</result>
-->     </test>

Environment

SDDC 5.x
ESXi 7.x/8.x managed by SDDC.

Cause

The issue can happen for following reasons:

If there are one or more vSAN disks are offline or corrupted. 

When a storage device that hosts ESXi partitions experience dirty bit leading vFAT partition corruption.

 

Resolution

There are multiple reasons why a vSAN disk can go offline. For more information in learning about disk status, the common reasons for disk status and to troubleshoot the issue, please visit Broadcom KB article 326859 

To identify and troubleshoot vFAT partition corruption issue, please follow the steps mentioned in Broadcom KB article 345227 

Additional Information

Note: Please raise a Support Case with vSAN team before proceeding further.