Unable to patch or upgrade an ESXi host as bootbank is pointed to /tmp

Products

VMware vSphere ESXi

Issue/Introduction

When attempting to patch or upgrade an ESXi host using VMware vSphere Lifecycle Manager (vLCM), the process fails. You will observe the following symptoms:

vLCM/vCenter Error:

"An error occurred during host configuration: /usr/sbin/esxupdate returned with exit status: 30."

Could not stage image profile '(Updated) DEL-ESXi_803.<build_number>':The transaction is not supported: VIB VMware_bootbank_gc_8.0.3-0.88.<build_number> cannot be live installed. VIB VMware_bootbank_esx-update_8.0.3-<build_number> cannot be live installed. VIB VMware_bootbank_esx-base_8.0.3-<build_number>cannot be live installed. VIB VMware_bootbank_vsan_8.0.3-<build_number>cannot be live installed. VIB VMware_bootbank_native-misc-drivers_8.0.3-<build_number>cannot be live installed. VIB VMW_bootbank_vmkusb_0.1-24vmw.<build_number>cannot be live installed. VIB VMware_bootbank_gc_8.0.3-<build_number>cannot be removed live. VIB VMware_bootbank_esx-update_8.0.3-<build_number>cannot be removed live. VIB VMware_bootbank_esx-dvfilter-generic-fastpath8.0.3-<build_number>cannot be removed live. VIB VMware_bootbank_esx-base_8.0.3-<build_number>cannot be removed live. VIB VMware_bootbank_native-misc-drivers_8.0.3-<build_number>cannot be removed live. VIB VMware_bootbank_vsan_8.0.3-<build_number>cannot be removed live. VIB VMW_bootbank_vmkusb_0.1-23vmw.<build_number>cannot beremoved live.

Error messages indicating that VIBs (e.g., esx-base, vsan, vmkusb) cannot be live installed or removed.
Running ls -l / shows the bootbank symlink pointing to a temporary directory in /tmp instead of a physical partition: bootbank -> /tmp/bootbank#######
The SD-Card or USB boot media is visible in the physical server console (iDRAC/iLO) and via ISO boot, but is missing from the ESXi command line (esxcli storage core device list).

hostd.log (/var/log/hostd.log) records the failure of the InstallV2 task with exit code 30:

YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099334]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 1352 : Could not stage image profile ' (Updated) DEL-ESX1 803.
24022510-A00': The transaction is not supported: VIB VMware_bootbank_esx-update 8.0.3-0.88.<build_number> cannot be live installed. VIB VMW_bootbank_vmkusb_0.1-24v
mw.803.0.88.<build_number> cannot be live installed. VIB VMware_bootbank_gc 8.0.3-0.88.<build_number> cannot be live installed. VIB VMware_bootbank_native-misc-drivers_8
be live installed. VIB VMware_bootbank_esx-base 8.0.3-0.88.2
5067014 cannot be live installed. VIB VMware bootbank_esx-dvfilter-generic-fastpath 8.0.3-0.79.24859861 cannot be removed live. VIB VMware_bootbank_esx-base
be removed live. VIB VMware_bootbank_vsan 8.0.3-0.79.24859861
be removed live. VIB VMW_bootbank vmkusb 0.1-23vmw.803.0.70.2467446439
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099338]: [Originator@6876 sub=Vimsvc. TaskManager opID-3f290071-da-24b5 sid-52220537 user-vpxuser:com. vmware. vcIntegri
ty] Task Completed : haTask-ha-host-vim.host. PatchManager. InstallV2-1886422344 Status error
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099338]: [Originator@6876 sub-Hostsvc. PatchManager opID-3f290071-da-24b5 sid-52220537 user=vpxuser:com.vmware.vcInteg
rity] PatchManager InstallV2 task returned: 30
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099336]: [Originator@6876 sub-Vimsvc.TaskManager opID-57986eal-87-24fl sid-52220537 user=vpxuser:com. vmware.vcIntegri
ty] Task Created : haTask-ha-host-vim.host. PatchManager. ScanV2-1886422367
YYYY-MM-DDTHH:MM:SS  In(166) Hostd[2099335]: [Originator@6876 sub-SysCommandPosix opID-57986eal-87-24fl sid=52220537 user=vpxuser:com.vmware.vcIntegrity]
ForkExec (/usr/sbin/esxupdate) 2115256
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099323]: [Originator@6876 sub=Hostsvc. VmkVprobSource] VmkVprobSource :: Post event: (vim.event. EventEx) {
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    key = 183,
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    chainId = -1702579664, STATTC39
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    createdTime = "1970-01-01T00:00:00Z",
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    userName = <user_name>
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    host - (vim.event. HostEventArgument) {
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    name = "<esxi_host_fqdn>",
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    host = 'vim. HostSystem:ha-host'
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    eventTypeId = "esx.audit.net.firewall.config.changed",
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    arguments = (vmodl.KeyAnyValue) [
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    (vmodl. KeyAnyValue) (
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    key = "1",
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    value - "enable"
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    (vmodl.KeyAnyValue) (
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    key = "2",
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    value = "esxupdate"

/var/log/esxupdate.log

The underlying cause is identified here—the installer cannot find the primary or secondary bootbank

YYYY-MM-DDTHH:MM:SS In(14) esxupdate[2115256]: Installer <class 'esximage. Installer. BootBankInstaller.BootBankInstaller'> was not initiated - reason: altbootbank is invalid: Error in loading boot.cfg from bootbank /bootbank: [Errno 2] No such file or directory: '/bootbank/boot.cfg'
YYYY-MM-DDTHH:MM:SS In(14) esxupdate[2115256]: Installers initiated are {'quickpatch': <esximage. Installer. QuickPatchInstaller. QuickPatchInstaller object at 0xb3d23970d0>, 'live': <esximage. Installer. LiveImageInstaller. LiveImageInstaller object at 0xb3dl1f61d0>}
YYYY-MM-DDTHH:MM:SS In(14) esxupdate [2115256]: runcommand called with: args = ['/sbin/smbiosDump'], outfile = None, returnoutput = True, timeout = 0.0.
YYYY-MM-DDTHH:MM:SS In(14) esxupdate [2115256]: Adding VIB VMware_bootbank_trx_8.0.3-0.88.<build_number> to ImageProfile (Updated) DEL-ESXi_803.<build_number>-A00
YYYY-MM-DDTHH:MM:SS Db(15) esxupdate [2115256]: VIBs ('VMware_bootbank_trx 8.0.3-0.79.<build_number>') are replaced by VIB VMware_bootbank_trx_8.0.3-0.88.<build_number>, removing them from ImageProfile (Updated) DEL-ESX1 803.<build_number>-A00

Inbuilt SDCard of 64 GB storage used in the ESXi is not seen by the host.
boot.log and jumpstart logs have been scanned and there is no reference of vmhba33 (SD-Card adapter), however the same is visible from iDRAC/iLO and also during upgrade process using ISO it is visible, but not seen within ESXi host.

Environment

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

This issue occurs when the vmkusb module is disabled on the ESXi host.

In modern ESXi versions, many internal SD-card readers and USB boot devices rely on the vmkusb driver to be detected. If this module is disabled, the hypervisor cannot claim the boot device during the "jumpstart" process. Consequently, ESXi mounts a temporary bootbank in RAM (/tmp), which lacks the persistence required for patching or upgrades.

Resolution

To resolve this, you must re-enable the vmkusb module and force the ESXi device manager to re-scan for the boot device.

1. Place the ESXi host into Maintenance Mode.

2. Run the following commands to enable the module and restart the device manager process:

esxcli system module set -m vmkusb -e true
kill -SIGHUP $(ps -C | grep vmkdevmgr | awk '{print $1}')

3. Check if the USB-attached boot devices are now visible:

lsusb
# or
esxcli storage core device list

4. Confirm the module is loaded and enabled:

esxcli system module list | grep vmkusb

Expected Output: vmkusb true true

5. Since the bootbank was pointed to `/tmp`, you must ensure the "Enabled" state of the module is saved to the physical storage once it is detected. Identify the active bootbank (usually `BOOTBANK1` or `BOOTBANK2`) and backup the state

cp /bootbank/state.tgz /vmfs/volumes/<ACTIVE_BOOTBANK_VOLUME>/