"VUM Remediation (installation) of an ESXi host failed" reported during patching ESXi host using SDDC Manager

Products

• VMware SDDC Manager
• VMware vSphere ESXi

Issue/Introduction

• Patching ESXi host using SDDC Manager may fail with error "VUM Remediation (installation) of an ESXi host failed"
• vSphere Client session reports the failure as "Cannot execute upgrade script on host"
  
  
  
  
• /var/log/vmware/vcf/lcm/lcm-debug.log (on SDDC Manager)

YYYY-MM-DDTHH:MM:SS DEBUG [vcf_lcm,<OP ID>,####] [c.v.evo.sddc.lcm.model.task.Stage,http-nio-127.0.0.1-7400-exec-12] Stage details: Stage(id=##:##, name=null, description=null, type=ESX_UPGRADE_VUM_STAGE_INSTALL_UPDATE, order=##, startTime=null, endTime=null, duration=null, status=FAILED, errorType=RECOVERABLE, errorCode=com.vmware.vcf.error.runtime.esx.update.vum.remediation.error.unexpected, errorDescription=
VUM Remediation (installation) of an ESXi host failed., remediation=null, resources=null, errors=null, notifications=null, subtaskId=###, subtaskType=UPGRADE, taskId=<Task ID>, taskType=com.vmware.vcf.lcm.upgrade)

• /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log (on vCenter Server)

YYYY-MM-DDTHH:MM:SS error vmware-vum-server[PID] [Originator@6876 sub=HU-Upgrader] [upgraderImpl ##] Script execution failed on host: <ESXi Management IP>)
YYYY-MM-DDTHH:MM:SS info vmware-vum-server[PID] [Originator@6876 sub=VciRemediateTask.RemediateTask{##}] [vciTaskBase 1496] SerializeToVimFault fault:
  (integrity.fault.HostUpgradeRunScriptFailure) {

     faultTypeId = "com.vmware.vcIntegrity.HostUpgradeRunScriptFailure",
       (vim.KeyValue) {
           key = "reason",
           value = ""

• vua log (/var/log/vua.log) on ESXi host gets rolled over and may not capture the same timestamp as the error under vmware-vum-server.log
• Attempting a --dry-run using ESXi offline bundle through ssh session to the ESXi host reports MissingComponentError. Refer to Patching ESXi host using Command Line

[root@esxi:/] localcli software profile update -d "/vmfs/volumes/<datastore>/VMware-ESXi-<offline bundle version>-depot.zip" -p <profile name> --dry-run
Errors:
 [MissingComponentError]
 Missing reserved components BCM-storcli_007.2613.0000.0000-02,Broadcom-bnxt-Net-RoCE_230.0.236.0-1OEM.800.1.0.20613240,Intel-icen_1.14.2.0-1OEM.800.1.0.20613240,Intel-igbn_1.12.0.0-1OEM.800.1.0.20613240,MRVL-QLogic-FC_5.4.83.1-1OEM.803.0.0.23710970,amsdvComponent_701.11.7.0.2-1,fc-enablement-component_800.3.9.0.30-1,ilo-driver_700.10.8.2.2-1OEM.700.1.0.15843807,ssacli2-component_6.40.6.0-8.0.0,sutComponent_800.5.2.0.5-0

Cause

This issue is reported if the components under "/var/db/esximg/reservedComponents" is missing/corrupt on ESXi host.

Resolution

This is a known issue with VMware vSphere ESXi when reserved Components are corrupted. Currently, there is no resolution.

Workaround

Upgrade the affected ESXi host using ISO via direct console. Refer to Upgrade Hosts Interactively