Getting "An unknown error occurred while performing the operation" when patching the ESXi Host from Update Manager

Products

• VMware vSphere ESXi

Issue/Introduction

• When patching the ESXi Host from Update Manager, the task fails during the staging process with error "An unknown error occurred while performing the operation".
  
  
• Per /var/run/log/vmkernel.log of the affected ESXi Host, both the bootbank and altbootbank partitions are VALID so both the partitions aren't broken or unhealthy. The same can also be confirmed using partedUtil and dosfsck utilities. Refer Using partedUtil command line disk partitioning utility on ESXi and "A problem with one or more vFAT bootbank partitions was detected", Corrupted vFAT partitions causing upgrade or pre-check failures for detailed instructions on how to check and confirm the health of vfat partitions on the ESXi Host.
  
   In(182) vmkernel: <7>BANK5: state=VALID build=<ESXi build and version> updated=1 quickboot=0 UUID=<ID>
 In(182) vmkernel: <7>BANK6: state=VALID build=<ESXi build and version> updated=2 quickboot=0 UUID=<ID>
 In(182) vmkernel: <7>BANK6: default boot bank.
 In(182) vmkernel: <7>EXEC: -R "/EFI/VMware/safeboot64.efi -r -m /EFI/VMware/mboot64.efi" -p 6 -c /boot.cfg -t "Loading VMware 
  
  
• The /var/run/log/lifecycle.log of the affected ESXi Host confirms that "Timeout" value of the bootbank's boot.cfg file should be a non-negative integer so the Patch Installer is not able to stage the host for the upgrade/update. 
  
  In(14) lifecycle[]: Downloader:630 Successfully executed cmd: localcli --plugin-dir /usr/lib/vmware/esxcli/int/ networkinternal firewall firewallRuleset set --ruleset-id esxupdate --service-name esxupdate --enabled false
In(14) lifecycle[]: HostImage:248 Installer <class 'vmware.esximage.Installer.BootBankInstaller.BootBankInstaller'> was not initiated - reason: altbootbank is invalid: Error in loading boot.cfg from bootbank /bootbank: Timeout must be a non-negative integer, not -1
In(14) lifecycle[]: HostImage:268 Installers initiated are {'quickpatch': <vmware.esximage.Installer.QuickPatchInstaller.QuickPatchInstaller object at 0x83d0e8e7d0>, 'live': <vmware.esximage.Installer.LiveImageInstaller.LiveImageInstaller object at 0x83d132bdd0>}
Er(11) lifecycle[]: imagemanagerctl:85 [KeyError]
Er(11)[+] lifecycle[]: 'boot'
Er(11) lifecycle[]: imagemanagerctl:90 Traceback (most recent call last):
Er(11) lifecycle[]: imagemanagerctl:90   File "/usr/lib/vmware/lifecycle/bin/imagemanagerctl.py", line 367, in software
Er(11) lifecycle[]: imagemanagerctl:90     t.ScanImage(swSpec, opts.depots, opts.taskId)
Er(11) lifecycle[]: imagemanagerctl:90   File "/lib64/python3.11/site-packages/vmware/esximage/Transaction.py", line 1984, in ScanImage
Er(11) lifecycle[]: imagemanagerctl:90     *_getPatcherFromImageSpec(imageSpec, depotUrls))
Er(11) lifecycle[]: imagemanagerctl:90   File "/lib64/python3.11/site-packages/vmware/esximage/Transaction.py", line 2937, in _getPatcherFromImageSpec
Er(11) lifecycle[]: imagemanagerctl:90     specMgr = _getSoftwareSpecMgr(imageSpec, depotUrls)
Er(11) lifecycle[]: imagemanagerctl:90   File "/lib64/python3.11/site-packages/vmware/esximage/Transaction.py", line 3056, in _getSoftwareSpecMgr
Er(11) lifecycle[]: imagemanagerctl:90     depotMgr = DepotMgr.DepotMgr(depotSpec, connect=True)
Er(11) lifecycle[]: imagemanagerctl:90   File "/lib64/python3.11/site-packages/vmware/esximage/ImageManager/DepotMgr.py", line 105, in __init__
Er(11) lifecycle[]: imagemanagerctl:90     self._connect(ignoreError=ignoreError, validate=validate,
Er(11) lifecycle[]: imagemanagerctl:90   File "/lib64/python3.11/site-packages/vmware/esximage/ImageManager/DepotMgr.py", line 189, in _connect
Er(11) lifecycle[]: imagemanagerctl:90     self._loadLocalMetadata()
Er(11) lifecycle[]: imagemanagerctl:90   File "/lib64/python3.11/site-packages/vmware/esximage/ImageManager/DepotMgr.py", line 203, in _loadLocalMetadata
Er(11) lifecycle[]: imagemanagerctl:90     profiles.append(hostImage.stagedimageprofile)
Er(11) lifecycle[]: imagemanagerctl:90   File "/lib64/python3.11/site-packages/vmware/esximage/HostImage.py", line 546, in stagedimageprofile
Er(11) lifecycle[]: imagemanagerctl:90     if self.installers[installer]:
Er(11) lifecycle[]: imagemanagerctl:90 KeyError: 'boot'
  
  

Environment

VMware vSphere ESXi 8.0.x
VMware vCenter Server 8.0.x

Cause

The mboot configuration file (boot.cfg) has a timeout value of 5. In case the host encounters a fatal error, it exits with the error status after TIMEOUT seconds. 

• Below is how it looks like under normal circumstances.
  
  bootstate=0
  title=Loading VMware ESXi
  timeout=5
  prefix=
  kernel=b.b00
  
  
• Below is how it might look for the problematic ESXi Host.
  
  bootstate=0
  title=Loading VMware ESXi
  timeout=-1
  prefix=
  kernel=b.b00

Resolution

1. Navigate to the bootbank of the affected ESXi Host.
   
   cd /bootbank
   
   
2. Take a backup of the boot.cfg file inside the bootbank (or altbootbank depending on where the file has an incorrect value).
   
   cp boot.cfg boot.cfg.bak
   
   
3. Use the vi editor to modify the file and change the timeout value from -1 to 5.
   
   
4. Save the configuration file and re-attempt the Host remediation.